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ABSTRACT 

The relationship between course work and general 
learning at Stanford University (California) , Mills College 
(California) , and Ithaca College (New York) was studied. The study 
used the Differential Course Work Patterns (DCP) Project faculty 
survey and Cluster Analytic Model (CAM) to link course work to 
student assessment. The study examined what the faculty perceptions 
of general learned abilities are in relation to the courses they 
taught, what advice faculty give to students who want to improve 
their general learned abilities, the results from the quantitative 
findings of the CAM study with the DCP for levels of congruency; and 
faculty members* perceptions of the courses they teach. Findings for 
both instruments indicated that reading comprehension, quantitative 
comparisons, and analytic reasoning were the types of general learned 
abilities where large changes occurred. At all three institutions 
students at or above the mean were more likely to enroll in course 
work associated with improvement in student learning. At all three 
institutions, faculty overestimated student cognitive skills in 
sentence completion and regular mathematics. Included are 32 
references, copies of a letter to prospective participants, a copy of 
the DCP survey instrument, and procedures for syllabi and test 
administration. (JB) 
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INTRODUCTION 

The capacity to determine which parts of the college curriculum contribute 
to gains in the general learning of undergraduates is of critical importance to 
higher education institutions. There is general agreement among researchers and 
practitioners that undergraduate study should foster certain essential skills, 
abilities and habits of mind among all students. Indeed, the requirement of the 
baccalaureate degree by many employers, most professional schools and almost all 
graduate study programs presumes a level or degree of development of general 
learned abilities that surpasses secondary education. The responsibility for 
developing these "general learned abilities" rests not only with general 
education programs, per se, but with the entire curriculum, including the 
academic major. In recent years many institutions, including 80 percent of 
four-year colleges, have initiated outcomes assessment programs at some level. 
Using a variety of instruments and criteria, these assessments measure such 
desired outcomes as the ability to interpret data, do basic computational 
mathematics, make comparisons using quantitative data, read a passage and 
interpret its themes, make deductive choices, analyze a piece of information and 
break it down into its component parts, recognize like and opposing concepts, 
and decipher the use of concepts and terms within a verbal context. The exact 
outcomes differ from test to test, those listed above being those measured by 
the General Test of the Graduate Record Examinations. 

Assessments of this kind are useful in documenting that learning has 
occurred. They cannot tell us, however, which courses most consistently produce 
gains in learning for specific groups of students over time at particular 
institutions. Such information would be extremely useful. Knowing the degree 
to which different courses contribute to different learning outcomes would 
provide a college or university with an empirical basis for curriculum review. 
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Knowledge of such links between coursework and learning could serve as a 
powerful source of information which would complement faculty wisdom, student 
evaluation, and other means of appraising the extent to which particular sets 
and sequences of courses have the effect for which they were intended. Such 
information could also be used to improve the academic advising and guidance 
students receive in making course selections (Ratcliff, 1990a, 19*0b, 1990c). 

Over the past 6 years we have developed a model for linking assessments of 
the general learning of undergraduates with the coursework in which they 
enrolled (Ratcliff, 1987, 1988, 1990a, 1990b, 1990c; Jones & Ratcliff, 1990a; 
1990b; Ratcliff & Jones, 1990; 1991). This research has proceeded under the 
rubric of the Differential Coursework Patterns (DCP) Project, and the model for 
linking coursework to student assessment has been referred to as the Cluster 
Analytic Model (CAM). Its development and testing was supported first by the 
Office of Educational Research and Improvement of the U.S. Department of 
Education. Subsequent qualitative validity studies of the GRE item-types, trend 
analyses of coursework patterns, and studies of the applicability of the model 
to curricular reform, assessment program development and academic advising has 
been supported by the Exxon Educational Foundation. The CAM has been tested at 
six institutions: Stanford and Georgia State Universities, and Clayton State, 
F- er green State, Mills and Ithaca Colleges. In addition, CAM has been applied 
o student reports of enrollment patterns and ACT COMP scores at the University 
of Tennessee-Knoxville (Pike & Phillippi, 1989). In all instances where 
students made choices from a wide selection of coursework, the CAM consistently 
discriminated that coursework most associated with large gains in general 

learned abilities. 

In the most common application of the CAM, assessment instruments were 
administered to graduating seniors, and the results of these post-tests were 
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compared with the results of corresponding pre- tests of the same students. Such 
well-known standardized instruments were used: the SAT, GRE, ACT and ACT COMP 
examinations, as well as the Kolb Learning Styles Inventory and 
locally-constructed measures of student-perceived course difficulty. In fact, 
it is a great strength of the Model and an asset that seems to enhance its 
acceptability to faculty, that it is not solely dependent on instruments 
supplied by external vendors. It can use a locally-developed instrument, 
tailored to particular needs and extensively employing local judgment. A 
college, for instance, might administer its own essay examinations to freshmen 
and seniors, and its own faculty might grade them holistically; so long as the 
final evaluation, and/or its subparts, can be translated to a numeric scale, 
this instrument would be entirely adequate for inclusion and use with the 
Cluster Analytic Model. 

Some faculty and academic leaders have been quick to reject the 
standardized measures of general learning, arguing that tests such as the SAT, 
GRE and ACT COMP are biased, are normed on a select group of students, measure a 
narrow band of lower-level learning, and the like. These arguments come despite 
the fact that most American colleges and universities use either the SAT or ACT 
tests in undergraduate admissions decisions, and the GRE, LSAT, MCAT and 
Miller's Analogies are widely used in graduate and professional scnool 
admissions. It is no small hypocrisy that the academy is comfortable using 
standardized testing as an indicator of prior student learning but rejects the 
notion that such testing is appropriate criteria in describing the effects of 
their college environment on their students. Proponents of qualitative measures 
of student learning often do not address the problems of assessing general 
learning in large research universities or community colleges, v>v re thousands, 
not hundreds of students need to be assessed to determine the impact of the 



crvIu^oL iona! program. 

A °ommon stumbling block in the development of an assessment program is 
that of what form of test or assessment information to use. Curricular 
reviewers, reformers and researchers quickly acknowledge that there is no clear 
conception of what constitutes general learning. Such recognition emerges 
regardless of whether it is the college curriculum or the various tests and 
assessment devices that are being examined, h college that attempts to reach 
consensus among its constituents on either general education goals or on the 
"best" measure of general learned abilities will foster heated discussion. But 
the quest for consensus on what should be the common intellectual experience of 
undergraduates may end in irresolution or, worse, abandonment of the assessment 
initiative. Instead of searching for the ideal measure of general learning in 
college, those charged with assessment can better direct their energies toward 
the selection of a constellation of assessment means and measures that appear to 
be appropriate criteria for describing one or more dimensions of the general 
learning goals of the college. 

The Cluster Analytic Model provides a basis for determining the relative 
extent to which each measure explains general student learning within a given 
college environment. When nine different assessment measures were used with the 
CAM, for example, we were able to determine what proportion of the variation in 
student scores was explained by each measure. This information can lead to a 
decision-point for the academic leader or faculty committee charged with the 
development and oversight of the assessment program, if a measure of general 
learning does not explain much of the variation in student scores, one option is 
to conclude that the measure is inappropriate to the students and the 
educational program of that particular college or university. In short, it can 
assist in the discard of that particular form of evaluation as superfluous and 




unnecessary. An alternative conclusion is that the institution is not devoting 
sufficient attention to the type of learning measured. Here, an examination of 
the assessment instrument itself relative to the curriculum is called for. 
Again, the Cluster Analytic Model can point to those courses and classes that 
were associated with gains in student learning on the measure in question. 

While we have tried to give you a brief overview of the Cluster Analytic 
Model, it is not our intention to present or justify its constructs, assumptions 
and procedures in this paper. And despite our enthusiasm for the CAM, we should 
note that it should still be regarded as an exploratory research technique. In 
this paper, we present research funded by the Exxon Educational Foundation 
wherein we used CAM to determine the relationship between coursework and general 
learning at a particular college. This information told us what, among the nine 
measures of learning we used, explained most of the gains in student learning. 
Then we sought to verify and to elaborate on that learning by talking with the 
faculty who taught the courses associated with such gains. In the end, we have 
found some interesting relationships between certain types of questions in the 
General Tests of the Graduate Record Examinations and the faculty's intentions 
in teaching selected courses that are part of the undergraduate curriculum of 
that institution. 



OBJECTIVES 

The aim of the differential coursework patterns faculty survey is 
two-fold. First, in Part One, the major goal is to determine what the taculty 
perceptions of general learned abilities are in relation to the courses they 
teach. Specifically, do the faculty perceive that their courses help students 
gain in different areas of their general learned abilities and how do their 
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courses enhance general learned abilities? In addition, this research explores 
what advice faculty give to students who want to improve their general learned 
abilities. In this research, general learned abilities denotes students 
cognitive development and is further defined by the criteria selected to measure 
general learning as described in the methodology section. 

A secondary goal of Part Dne is to compare the results from the 
quantitative findings of the CAM study with the qualitative procedures used in 
recent faculty interviews to determine the level of congruency of these two 
analyses concerning the courses effects in the general learned abilities of 
students. In addition to the interviews, we collected examples from the course 
syllabi and final examinations to provide corroborating evidence for each 
interview. 

The second aim of the DCP faculty survey, discussed in Part Two of this 
report, is to assess faculty members' perceptions of the courses they teach. 
More specifically, we examine issues such as how students are evaluated, the 
desired outcomes of class teaching, course planning, and class scheduling. 

FKAMEMORK 

A literature review indicated that no single curricular model and no single 
analytical process clearly identified the effect of coursework patterns on the 
general learned abilities of students. Ratcliff and associates (1987) developed 
the Cluster Analytic Model to determine the effect of coursework in colleges and 
universities on the general learned abilities of undergraduates. The CAM has 
demonstrated strong secondary validity and reliability within the context of a 
variety of higher education institutional types and student populations 
(Ratcliff, 1988, 1990c). The model uses a conceptual-empirical approach. 
Student decisions about courses and actual selections indicated on their 
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transcripts guided the empirical search for coursework patterns associated with 
gains in general learned abilities. 

This model guided the development of an interview protocol used to 
ascertain faculty perceptions concerning their courses relationship with the 
different areas of student general learned abilities. 

OVERVIEW OF METHODOLOGY AND PROCEDURES 

The procedures followed the Cluster Analytic Model. While incoming student 
ability of the sample was controlled by SAT scores, the exiting student 
achievement was measured by the Graduate Record Examination (GRE) scores. 
Specifically, the residual differences from the predicted and observed scores on 
the nine item- types within the General Test (of the GRE) served as the measures 
of exiting student achievement. In the Verbal section of the GRE, the four 
item- types are Analogies, Sentence Completion, Reading Comprehension, and 
Antonyms. Analogy items test students' ability "to recognize relationships 
among words and the concepts they represent and to recognize when these 
relationships are parallel. The process of eliminating four wrong answer 
choices requires one to formulate and then analyze the relationships linking six 
pairs of words" (ETS, 1988, p. 28). Antonym items provide a direct test of the 
student's vocabulary. However, the purpose of this item- type is not merely to 
measure the student's vocabulary, but also to gauge "the student's ability to 
reason from a given concept to its opposite" (ETS, 1988, p. 29). For the 
Reading Comprehension items, students must read narrative with "understanding, 
insight and discrimination." These passages challenge a student's ability to 
analyze using a variety of perspectives "including the ability to recognize both 
explicitly stated elements in the passage and assumptions underlying statements 
or arguments in the passage as well as the implications of those statements or 
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arguments" (ETS, 1988, p. 31). Due to the length of the narratives around which 
the questions for this item-type are built, students are given ample opportunity 
to assess a variety of relationships, such as the function of a key word in a 
passage, the relationships among several ideas, or the relationship of the 
author to the topic or the audience. Sentence Completion items determine the 
student's ability to "recognize words or phrases that both logically and 
stylistically complete the meaning of a sentence" (ETS, 1988, p. 30). The 
student must decide which of five words, sets of words or phrases can best 
complete a sentence. In completing this type of task, the student must consider 
which answer gives the sentence a logically satisfying meaning and stylistically 
integrated whole to the discourse. 

In the Quantitative section of the GRE the item-types are Quantitative 
Comparison, Regular Mathematics, and Data Interpretation. The Regular 
Mathematics item-type has also been labelled Discrete Quantitative questions and 
Arithmetic, Algebra and Geometry in various GRE and ETS publications. The 
Quantitative Comparisons items test the student's ability "to reason quickly and 
accurately about the relative sizes of two quantities or to perceive that not 
enough information is provided to make such a decision" (ETS, 1988, p. 34). 
Data Interpretation items present sets of data in graphs and tables and ask 
students to synthesize the information, choose the correct data to answer the 
question, or to determine that the information needed is not present in the data 
set . 

In the Analytic section, the item- types are Analytical Reasoning and 
Logical Reasoning. Analytic Reasoning items measure a student's ability "to 
understand a given structure of arbitrary relationships among fictitious 
persons, places, things, or events, and to deduce new information from the 
relationships" (ETS, 1988, p. 38). Logical Reasoning items assess a student's 



ability to understand, analyze and evaluate positions and contentions. Specific 
questions may evaluate a student's ability to recognize a point of argument or 
the assumptions on which a position is based, to draw conclusions or form 
hypotheses, to assess the manner of arguments and the evidence supporting 
them. 

While the GRE General Tests are designed to describe the student's broad 
verbal, mathematics and analytic abilities, the nine individual item-types of 
the Test provide discrete measures of general learned abilities. These nine GRE 
item-type residual scores represented the gains students experienced in general 
learned abilities from the time they entered college to the time of GRE testing 
during their senior year. 

Next the coursework patterns for each sample were identified among the 
student transcripts. The unit of analysis was a single course. Each course 
examined had nine attributes represented by the nine residual item- type scores 
of students enrolling in the course. Courses with sufficient enrollment by the 
student sample were grouped according to the collective item-type scores of the 
students enrolling in the course. Therefore, each course had a mean residual 
score for each item-type. The effect of individual courses on test score 
residuals were determined by using cluster analysis. The cluster analysis 
techniques facilitated the construction of a classification scheme for 
unclassified data sets and it empirically examined the college curriculum using 
student decision-making behavior (represented on the student transcripts) as the 
primary source of information. Through the cluster analytic model, sets of 
courses were identified that were associated with gains in general learning on 
one or more of the nine (item- type) areas of general learning. 

Using this course data from the samples, instructors whose individual 
courses yielded high positive means on the GRE item-types and enrolled three or 
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more students were identified. Campus liaisons (at Stanford, Ithaca, and Mills) 
sent a letter describing the research project to these instructors (see Appendix 
A) . Then the researchers phoned each professor to schedule a forty minute 
interview. At Stanford, the Registrar and Senior Associate Provost's office 
initially coordinated the interview schedule. However, due to the low response 
rate, subsequent site visits and interview schedules were coordinated by the 
researchers. Two site visits (usually a week long) were made to Ithaca College 
and Stanford University. Due to the changes in the administration and 
controversies relative to mission, faculty interviews were not scheduled for 
Mills College until April 1991. Results from Mills College have been 
incorporated into this report. 

An interview protocol was utilized to assess faculty perceptions. This 
protocol was pilot tested at two Midwestern universities and two Midwestern 
colleges to insure the clarity of questions. Each professor was asked if the 
specific course he or she taught aided students in answering questions similar 
to each of the nine GRE item-types. Faculty were given examples of the GRE 
item-type questions (taken directly from a Graduate Record Examination) so that 
they could make direct comparisons. If a professor indicated that his or her 
course was helpful, then a follow-up question was asked concerning how the 
course aided students. Additional questions probed how faculty decide what 
advice to give students who want to develop their abilities in a specific area 
and what courses the faculty would recommend for students who wished to improve 
their abilities. Finally, course syllabi and final examinations were collected 
and analyzed as supplemental sources of information regarding the purposes of 
the courses and their means of evaluation. 

Two random samples of graduating senior 3 were drawn from Stanford 
University. The first sample consisted of 105 students who constituted 
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approximately 8 percent of the graduating seniors during the 1987-1988 academic 
year at Stanford University. The second sample of 161 students were graduating 
seniors from the 1988-1989 academic year. 

Three random samples of graduating seniors were drawn from Ithaca College. 
The first sample consisted of 146 students who constituted approximately 14 
percent of the graduating seniors during the 1987-1988 academic year at Ithaca 
College. The second sample of 191 students constituted 19 percent of the 
population of graduating seniors during 1988-1989 academic year. A third sample 
of 186 students constituted 16 percent of the graduating seniors during the 
1989-1990 academic year. 

Two random samples of graduating seniors were drawn from Mills College. 
The first sample consisted of 62 students who were graduating seniors from the 
1987-1988 academic year. The second sample of 44 students were graduating 
seniors from the 1988-1989 academic year. The samples at these three 
institutions were proportional to the distribution of Scholastic Aptitude Test 
(SAT) scores, majors, and other socioeconomic characteristics of the population 
of graduating seniors at each institution (Ratcliff, 1990b; 1990c; 1990d) . 

FINDINGS OF DCP FACULTY SURVEY: PART ONE 
Stanford University Samples and Dominant Item-Types 

A cluster analysis of coursework and assessment scores for the Stanford 
samples produced a list of 94 courses where students had demonstrated high 
positive gains on one or more of the GRE nine item- types. The faculty who 
taught these courses were identified. From this group, 79 faculty agreed to be 
interviewed individually to assess their perception of their courses in relation 
to the general learned abilities of college students. Some faculty interviewed 
taught more than one course associated with improvement in student learning and 



responses were obtained for both courses. 

T or Stanford Sample #1, five item- types were dominant and associated wit* 
student improvement . Forty percent of the variation in Sample #1 test scores 
was explained by the Analytic Reasoning item- type. Another twenty percent was 
explained largely by Reading Comprehension, However / only one course was 
associated with this item-type in the interviews and therefore no further 
analysis was conducted with this item- type. Thirteen percent of the variation 
in Sample #1 was explained by Quantitative Comparisons / nine percent was 
explained by Antonyms / and seven percent was explained by Regular Mathematics. 

For Sample #2, thirty percent of the variation in their test scores was 
explained by Analytical Reasoning and this was comparable to the results in 
Sanqple #1. Another twenty-five percent was explained largely by the Antonyms 
and Quantitative Comparisons item-types. Another eleven percent was explained 
by Analogies and eight percent was explained by the Regular Mathematics 
item-types (Ratcliff, 1990a). 

The following discussion represents an aggregation of the data across these 
two samples since improvements in student learning were associated with some 
similar item- types. Tables 1 through 5 indicate the Stanford University courses 
represented in the faculty interviews and their association with the 
item-types. These relationships were identified from the quantitative 
procedures utilized in the Cluster Analytic Model. 
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Table 1 

Courses Represmted in Faculty Interviews at Stanford ftaivarsity 
Associated With igt u o w ma nt in Analytical Reasoning 



Course Rinbar Title Course ;hssbar Title 
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PSYCH 


n 1 c 

lib 


Social Development 


CHHI 


31 


Chemical Principles 


PSYCH 


lib 


T> ■ i rati I ■ ^ n i in mJT L^^Mtfl 

psychology ox men 


OfflW 


33 


Structure and Reactivity 


PSYCH 


1 in 
13b 


anooMai rsycnoiogy 




35 


Organic Haoafunctional Cocyoupds 


STAT 




T„i l__.t *r> Ot- at \ \ r*o 1 IMVi/vic T 

inutXBiCviOD \o yvCviavicai ibikhb a. 


CHffl 


36 


Qwaical Separation 


STAT 


1 1 1> 
110 


Statistics Hetboos in Engineering ana 


CHKH 


131 


Organic Polyf actional Compounds 






Physical Sciences 


CHEH 


135 


Physical Cbadcal Principles 


STAT 


116 


Tbeory Pronanlnty 


CUS LIT 


11 


Age of Heroes 


VTSS 


115 


Technology and Aesthetics 


Ecxa 


1 


Eleoentary Eooncsdcs 








ECCW 


51 


Ecaocnics Analysis I 








ECGB 


165 


International Econaaics 








EEC 


31 


Decision Analysis 








GE 


10 


Allied ftecbanics: Statistics 








GE 


30 


Engineering Thenaodynanics 








GE 


40 


Basic Electronics 








GE 


50 


Introductory Science of Materials 








GE 


60 


Engineering EoqpcbiIcs 








GEOL 


1 


Interpreting the Earth 








HIST 


1 


Europe: Antiquity , Middle Ages, and Renaissance 






HIST 


2 


Europe f roc the Mars of Religion to the Nation 










State 








HIST 


3 


Europe: 1815 to the Present 








ROM 


62 


Western Thought and Literature 








HUH BIO 


2 


Genetics, Evolution, and Ecology 








HOT BIO 


4 


The Hunan Organise 








IE 


60 


Analysis of Production and Operation Systems 








IE 


100 


Organisations: Theory and Management 








MATH 


20 


Calculus and Analytic Gecnetry 








HATH 


41 


Calculus/Ecouttics 








HATH 


130 


Ordinary Differential Equations 








HE 


103 


Manf actoring Technology 








HE 


111 


Stress, Strain, and Strength 








HE 


112 


Mechanical Systens 








HE 


113 


Engineering Design 








HE 


131 


Fluid Mechanics 








HE 


161 


Mechanical Vibrations 








OPE* RE 


151 


Introduction to Operations Research 1 








OPES RE 


152 


Introduction to Operations Research I 








POL SCI 


1 


Major Issues in American Public Policy 








POL SCI 


182 


Introduction to American Lav 








PHYS 


21 


Mechanics and Heat 








PHYS 


61 


Advanced Freshaen Physics 








PHYS 


62 


Advanced Freshsen Physics 








PHYS 


63 


Advanced Freshaen Physics 








PSYCH 


106 


Cognitive Psychology 
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Trtble 2 

Courses Represented in Faculty Interviews at Stanford University 
Associated With Improvement in Analogies 



Course 


Number 


llT.i.6 


ART H 


10 


Introduction to Art 


CIAS LIT 


8 


Classics Politics 


COMM 


108 


Mass Communication Theory 


COMM 


110 


Media and law 


COMM 


170 


Coircnunication and Law 


COM SCI 


108 


Computer Science and Fundamentals 


POL SC 


35 


International Politics 


VTSS 


115 


Technology and Aesthetics 



Table 3 

Courses Represented in Faculty Interviews at Stanford University 
Associated With Improvement in Antonyms 



Course Number Title 



ANTH 


3 


Human Prehistory 


ART H 


10 


Introduction to Art 


BIO SCI 


31 


Molecular and Cellular Biology 


BIO SCI 


32 


Development and Organism Biology 


BIO SCI 


33 


Evolutionary Biology and Ecology 


BIO SCI 


166 


Genet ics 


GE 


30 


Engineering Theiodynamics 


GE 


60 


Engineering Economics 


HUM BIO 


32 


Not Available 


MATH 


130 


Ordinary Differential Equations 


ME 


33 


Introductory Fluids Engineering 


ME 


103 


Manufacturing Technology 


ME 


111 


Stress, Strain, and Strength 


ME 


112 


Mechanical Systems 


ME 


113 


Engineering Design 


ME 


131 


Fluid Mechanics 


PHYS 


55 


Light and Heat 
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Table 4 

Courses Represented in Faculty Interviews at Stanford University 
Associated With Improvement in Quantitative Comparisons 



Course 


Number 


Title 


ART H 


10 


Introduction to Art 


BI SC 


42 


Not Available 


CIAS LIT 8 


Classics Politics 


CONN 


108 


Mass Communication Theory 


COMM 


110 * 


Consnunication Media and tne uaw 


COMM 


170 


Consnun icat ion ana lji i 10 


COM SCI 


108 


Computer Science and Fundamentals 


HUM BIO 


10 


Human Sexuality 


MATH 


41 


Calculus and Analytic Geometry 


POL SC 


35 


International Politics 


POL SC 


116 


European Policy and Society 


SOC 


121 


Not Available 


VTSS 


110 


Philosophical and Ethical Issues in Public 






Policy 


VTSS 


115 


Technology and Aesthetics 














Table 5 






Courses 


Represented in Faculty Interviews at Stanford University 


Associated With Improvement in Regular Mathematics 


Course 


Number 


Title 


RNGR 


35 


Not Available 


HUM BIO 


4 


The Human Organism 


MATH 


21 


Calculus and Analytic Geometry 


MATH 


41 


Calculus and Analytic Geometry 


VTSS 


110 


Philosophical and Ethical Issues in 






Public Policy 



Faculty Responses 

Many faculty at Stanford believed their courses helped students gain in 
their general learned abilities. As Table 6 indicates, the majority of faculty 
(78.7%) perceived that their courses helped students to improve in Logical 
Reasoning, followed by 64.9% of faculty who believed their courses helped 
students in Analytical Reasoning, and 67.0% viewed their courses as beneficial 
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in the area of Data Interpretation. Reading Comprehension was considered by 
61.1% of the faculty as an area where their courses would aid students to 
iiqprove . 



Table 6 

Faculty Perceptions of General Learned Abilities at Stanford University 



CRR ITEM-TYPES . 

Sentence Reading Regular yuaatitative Data Logical Analytic 

Coaptation Analogies Catprebenaion Antonyns Hathentics Cocparisons interpretation Seasoning Be—fla? 



By course helps 


31.91% 


37.23% 


61.70% 


24.47% 


35.11% 


44.68% 


67.02% 


78.72* 


64.89% 


students inprove 


(n=30) 


(n=35) 


(n=58) 


(d=23) 


(n=33) 


(n=42) 


(n=63) 


(n=74) 


(D=61) 


My course does 


64.89% 


59.57% 


36.17% 


73.40% 


57.45% 


53.19% 


30.85% 


19.15% 


34.04% 


not help 


(n=61) 


(n=56) 


(n=34) 


(n=69) 


(n=54) 


(n=50) 


(n=29) 


(n=18) 


(n=32) 


students kprove 




















Don't know 


3.19% 


3.19% 


2.13% 


2.13% 


7.45% 


2.13% 


2.13% 


2.1 


1.08% 




(n=3) 


(n<J) 


(n-2) 


(c=2) 


(o-7) 


(D=2) 


(n=2) 


(n-2) 


(n=l) 



The faculty responses of whether they considered their own courses as 
enhancing a student's ability to respond to each of the nine item-type areas 
were compared with the results from the quantitative analysis. Recall that 
through the Cluster Analytic Model, courses were identified and their 
associations with the nine measures of general learning. Table 7 indicates that 
low levels of congruency existed between the faculty responses and the 
quantitative results primarily for the Analogies , Antonyms, and Quantitative 
Comparisons item- types* Three reasons account for some of these differing 
viewpoints- First, many faculty believed that the academic level of students 
attending Stanford University was very high. Therefore, they perceived that 
students would have already attained solid levels of abilities in these areas 
prior to college attendance. They believe their own courses would not enhance 
these abilities. However, the quantitative analysis indicated that there were 

OK 
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many courses associated with student improvement in these areas. The second 
reason for the incongruence of results between the faculty perceptions and tho 
quantitative analysis involve the actual interview instrument used. Faculty 
were given examples of the item-types taken directly from a Graduate Record 
Examination. They often based their remarks on the content of the specific 
question rather than the broader goal of what ability the item-type was 
measuring. For example, in the area of Quantitative Comparisons, faculty were 
given a test item with a geometric figure. They viewed this question as dealing 
with geometry and were unable to see connections with their own courses. 
However, Quantitative Comparisons items were intended to measure the student's 
ability to make decisions about the sizes of two different quantities or to 
determine if there was not enough information to make a decision (ETS, 1988, p. 
34). A third reason for the lack of congruence may be that these item-types did 
not explain the majority of score variance among students in either sample. 

There were higher levels of agreement for the Regular Mathematics and 
Analytical Reasoning item-types. More faculty believed their courses helped 
students to improve in these areas and the quantitative results supported their 
perceptions. Overall, faculty seemed to better understand what abilities these 
item-types were measuring. Also, Analytic Reasoning explained the largest 
proportion of score variance among students (Ratcliff, 1990a). 



*~ b 
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Table 7 

Level of Agreement Between Responses from Faculty Interviews and 
Quantitative Analysis for Stanford University 

GRE ITEM-TYPES 



Regular Quantitative Analytic 
Analogies Antonyms Mathematics Comparisons Reasoning 



Percentage of 
agreement that 

course helps to 14.28% 31.25% 50.00% 20.00% 66.67% 

improve learned 

ability 



Faculty at Stanford University who responded that their courses helped 
students improve their abilities in relation to the GRE item-types were asked 
how their individual courses helped. They were also asked what courses to 
recommend for students to take if they wanted to improve their abilities. 
Specific examples from the faculty examinations supplement the faculty interview 
responses in reference to the five major item-types of Analytical Reasoning, 
Quantitative Comparisons, Antonyms, Analogies, and Regular Mathematics. 



Analytical Reasoning 

Some faculty believed that their courses helped students to improve in 
Analytical Reasoning, A biological science professor stated that in his course 
students were regularly given data and had to visualize it and interpret it 
based upon some information that was given to them. In a computer science 
course, a professor said he gave students complicated programs which required 
skills in abstraction and managing complexities . Students in this course were 
required to determine what information was true before they could work on the 
programs and had to fit many complex pieces of information together. Overall, 
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the professor believed his course required vigorous analysis. 

A chemistry professor's test item exemplifies the parallels we found with 

Analytical Reasoning types of abilities. 

In 1955, a small asteroid landed in the middle of San Francisco bay, 
ending the drought in surrounding counties. By 1998, surviving 
scientists had analyzed the cooling mineral debris for traces of 
organic matter. They discovered an unusual protein that they named 
Dilsegin (for Diluvio Segundo). 

Dilsegin has the following properties: 

1. molecular weight 72,500; 

2. does not form a phenylthiohydantoin when treated with Edman 
regent; 

3. does not react with diazome thane; 

4. is not cleaved to smaller peptides by trypsin or chymotrypsin; 

5. complete hydrolysis yields an equimolar mixture of Gly, Phe, Ala, 
Lys, and His, 

* What do clues 2 and 3 tell you about the structure of Dilsegin? 

An engineering economy course also contained a test item illustrating the 
similarity to an Analytical Reasoning item- type; 

Given a lottery, we know the following regarding John, Jim, and Larry. 

1. John has a certain equivalent of $90 for the lottery. 

2. Jijn is risk neutral and has a certain equivalent of $100 for the 
lottery. 

3. Larry has a certain equivalent of $80 for the lottery. 

* Which of the following is true? 

a. John is risk seeking; Larry is more risk seeking than John. 

b. John, Jim, and Larry are risk neutral. 

c. John is risk-averse; Larry is more risk-average than John. 

d. John is risk seeking but Larry is risk-averse. 
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Faculty believed their own courses helped students to improve in Analytic 
Reasoning. Concrete examples from actual faculty examinations indicated links 
between coursework and the faculty's evaluation of a student's ability to 
understand a structure of relationships to deduce information (ETS, 1988). 

Quantitative Comparisons 

Some faculty believed that their courses helped students to improve in 
Quantitative Comparisons. An engineering professor had students do problem 
solving by looking at two alternatives in financial decisions and deciding which 
option was best. Students had to structure problems and reach solutions. A 
human biology professor stated in a more general manner that his course taught 
students how to systematically approach and disentangle complex problem. 
Overall, in this area faculty were unable to clearly see connections with their 
courses helping in Quantitative Comparisons . Part of the reason for this 
difficulty was that the actual item in the interview instrument was a geometric 
figure and faculty tended to concentrate on this visual image rather than 
focusing on the actual abilities associated with Quantitative Comparisons. 

An introductory science materials course exam contained a good example of 
Quantitative Comparisons item that asked students to compare the relative sizes 
of two different quantities. 

Indicate whether the following statement is true or false. 

Consider two metals, one with a melting point about 1600 C and one with 
a melting point of 600 C. At 500 C it is likely that the latter would 
have the higher vacancy concentration. 

An engineering economics course test item also provides a good example of 

Quantitative Comparisons where students must compare the results of two 

different gambles. 

Vou are given the option of two mutually exclusive gambles. 

For the first gamble, a coin will be flipped three times. If the 
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coin come up "heads" each of the three times, you will be given 
$1600 in five years. If "tails" comes up each of the three times, you 
will be given $800 in five years. For any other combination you will 
will be given nothing. 

For the second gamble, a coin will be flipped once. If the coin comes 
up "heads", you will be given $400 right away. If the coin cooes up 
"tails", you will be given nothing. 

If you bet so as to maximize expected value, which of the gambles should 
you take? Show why? 

Faculty viewed definite relationships between their own courses and the 
improvement of a student's ability to deal with Quantitative Comparisons. The 
faculty examinations provided concrete evidence that these abilities are 
assessed in coursework. 
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Regular Mathematics 

Some faculty believed that their courses helped students to iaprove in 

Regular Mathematics- A human biology professor stated in a general manner that 

his course taught students how to use statistics to describe a population. 

An automotive technology course exam provides an example of an item 

similar to Regular Mathematics. 

Assuming that the average energy requirements are 180 watt-hours per 
mile for a vehicle , calculate the weight of sodium-sulfur batteries 
required for it to achieve a 200 mile range. Assume that the efficiency 
of transferring energy from the battery to the wheels is 85%. 

Faculty believed their own courses helped students to improve in Regular 

Mathematics. Faculty examinations indicated that they did assess a student's 

ability to handle mathematics. 



Some faculty believed that their courses helped students to improve in 
Analogies. A professor in his history course used the primary activity of 
analysis of arguments and how elements related with each other* He believed 
that through this analysis in his course, students increased their vocabulary. 
An engineering professor believed his course taught new terms and concepts in 
the field so that students could understand current articles. 

A test item from a professor's course in psychology in a general way 
relates to Analogies and the notion of recognizing relationships among words and 
the concepts they represent . 



Analogies 



Which of the following observations best illustrates the idea that 
families need to be thought of as true social systems? 



a. 



A young boy establishes a secure attachment with his mother. 



Mothers are less inclined to play with their children when 
fathers are present* 
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c. Parents today are having fewer children. 

d. In today's families, an ever- increasing percentage of mothers 
choose to work outside the home. 

e. Abusive parents were often abused children themselves. 

In general, though, faculty did not have a ready sense of how students develop 
the ability to draw analogies. It was difficult to find examples of Analogies 
in the faculty examinations. 

Antonyms 

Some faculty believed that their courses helped students to improve in 
Antonyms. A professor in his communication course encouraged students to read a 
lot and learn vocabulary in the course. In a similar fashion an engineering 
professor believed his course taught students the specific acquisition of new 
terms and an understanding of them. 

A sociology professor's exam contains an item related with Antonyms where 
students had to reason with two opposing situations. 

Choose ONE of the following pairs of situations from the episodes 
recorded in your journals: 

A. Situations in which you were a high interactor and situations in 
which you were a low interactor. 

B. Situations in which there was a formal organization and formal 
roles and situations that were informal. 

C. Situations in which all of the participants were of the same gender 
and situations in which there were both males and females. 

Faculty believed their courses helped students to improve in Antonyms. However, 

it was difficult to find examples of this area in the faculty examinations. 

In general, faculty thought that the courses they taught enhanced students' 

abilities to answer certain GRE item-type examples. With the exception of 

Quantitative Comparisons, faculty selected those item examples which correspond 

with the types of student learning gains associated with their course. Analytic 
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keasoning produced the most score variance of the item-types in both Sample 
groups and was frequently picked by faculty as associated with learning 
occurring in their courses. 

Implications for Academic Advising at Stanford University 

Faculty were asked what courses to recommend for students to take if they 
wanted to improve their abilities as defined by the item-type areas of the GRE. 
In general, faculty did not have a consistent knowledge base for making such 
decisions. They drew upon their own undergraduate experience (usually at 
another college or university), The Stanford Bulletin , word of mouth 
recommendations of courses by other faculty or students. Faculty thought that 
courses in logic, computer science, engineering , philosophy, and biology helped 
students improve their abilities in Analytical Reasoning. Some faculty 
recommended these general areas while other faculty recommended specific courses 
offered at Stanford University which were usually courses in their own 
disciplines. Faculty were less knowledgeable about recommending courses outside 
of their disciplines. Since the majority of faculty were associated with 
technical and science related disciplines, these types of courses were 
frequently recommended • However, the cluster analysis indicated that other 
specific courses at Stanford in disciplines such as economics, history, 
psychology, political science, and statistics were associated with student 
improvement in Analytical Reasoning as well. 

Faculty thought that courses in mathematics, geometry, logic, and computer 
science helped students improve their abilities in Quantitative Comparisons. 
Again, faculty recommended specific courses offered at Stanford within their 
own disciplines or general traditional subject areas. However, the cluster 
analysis indicated that other specific courses in political science and 



humanities were associated with student improvement in Quantitative 
Comparisons. 

Faculty thought that courses in mathematics , geometry , logic, statistics, 
and engineering helped students improve their abilities in Regular Mathematics 
and these were the predominant courses in the CAM. Faculty thought that 
coursework in English, philosophy, history, fine arts, and liberal arts courses 
would help students improve their abilities in Analogies. Faculty often 
recommended these general areas- However, the cluster analysis indicated that 
other specific courses in computer science and communications were associated 
with student improvement . 

Faculty thought that courses in English, public speaking, literature, 
history, and foreign language would help students iinprove their abilities in 
Antonyms. Most faculty recommended these general areas. However, the cluster 
analysis indicated that other specific courses in biological science, 
engineering, and physics were associated with student improvement . This 
research demonstrated that there were many different types of courses and 
disciplines associated with improvement in student learning. Faculty 
were tentative in recommending general traditional subject areas. When faculty 
viewed direct linkages of their own discipline with a certain item-type, they 
were comfortable in suggesting specific courses in their own discipline. 

Ithaca College Samples and Dominant Item-Types 
A cluster analysis of coursework and assessment scores for these samples 
produced a list of 107 courses where students had improved on one or more of the 
GRE nine item-types. The faculty who taught these courses were identified. 
From this group, 93 faculty agreed to be interviewed individually to assess 
their perception of their courses in relation to the general learned abilities 



of college students. Same faculty interviewed taught more than one course 
associated with improvement in student learning and responses were obtained for 
all their courses that had been identified in the research. 

For Ithaca Sample #1, five item- types were dominant and associated with 
student improvement. Forty six percent of the variation in Sample #1 test 
scores was explained by Analytic Reasoning and Regular Mathematics item- types. 
Another twenty percent was explained largely by Quantitative Comparisons . 
Fourteen percent of the variation in Sample #1 was explained by Quantitative 
Comparisons, and eight percent was explained by Antonyms (Ratcliff, 1990b). 

For Sample #2, forty-eight percent of the variation in their test scores 
was explained by Analytical Reasoning and this was comparable to the results in 
Sample #1. Another seventeen percent was explained largely by the Reading 
Comprehension item-type. Another sixteen percent was explained by Quantitative 
Comparisons (Ratcliff, 1990b). 

For Sample #3, twenty-eight percent of the variation in their test scores 
was explained by Analytic Reasoning. Another seventeen percent was explained by 
the Antonyms item- type. Another seven percent was explained largely by Reading 
Comprehension and Regular Mathematics (Ratcliff, 1990c). 

The data and discussion represent an aggregation of the data across these 
three samples since improvements in student learning were associated with some 
similar item-types. Tables 8 through 12 indicate the Ithaca College courses 
represented in the faculty interviews and their association with the 
item-types. These relationships were identified through the quantitative 
procedures in the Cluster Analytic Model. The predominant item-types associated 
with student improvement were Analytical Reasoning, Reading Comprehension, 
Regular Mathematics, Quantitative Comparisons, and Antonyms. 




Table 8 

Ithaca Courses Targeted In Faculty Interviews and the CM Linkage 



to Sains in analytical Reasoning 








Coirse 


Hunger 


Coarse Title 


Coarse 


stater 


Pjnirm *HtAe 

i 1 ni i rrTT uuv 
















AC 


105 


Principles of Acct I 


SRA 


102 


nOQlC&LL rBXiSlJjQJA/yj 


IC 


105 


Principles of Acct I 


HO ED 


llu 


WCWflD w wwraie 


&HS 


205 


Critical Health Issues 


PHIL 


151 


Reasoning I 


IMS 


401 


Public Health and Epideoiology 


PHYSED 


81 


nOQ s roouxLu. 


MTTHRO 


104 


Cultural Anthropology 


PRISED 


125 


Hasan RnaxooY 


BIO 


101 


Fuodaaentals of Bio 


PHYSED 


12b 


It- mm lm n i m rrt MUD T jClVl 

nuoan ADuvcniy Iju-j 


BIO 


104 


Envtromental Biology 


PHYSED 


23/ 




B10 


112 


Food and Fed Control 


PHYSED 


zw 


MOv. BVol MttUg 


BIO 


201 


Anatcny and Physiology 


PHYSED 


334 


PnyslalogY 


BIO 


202 


Anatooy and Physiology 


PHYSED 


335 


Physiology 


CHEU 


102 


Conteaporary Chanlcal Issues 


PHYST 


103 


T_.i iVuit tjxn UaaI *K nr-nFoKMil ftfio 

XnXroQnCLiOD vQ neaXU) rruibttiuuiia 


CHS! 


111 


Fuadaaentals of Chea 


PHY5T 


203 


jjjiro io rnysicai. luerapy 


CHEH 


112 


Organic & Biochemistry 


PHYST 


311 




CHEM 


113 


Cheoistry Lab 


PHYpt 


331 


ttlSvOJluyy « raiiflni^jj 


COB 


102 


Career Development 


PHYST 


342 


M tnwnf rrt<T Q .1iiiit«wk 

Benroanatcoay « Heuropc 


csa 


110 


Introduction to Data Processing 


PHY5 


101 


Intro to rnysics 1 


csa 


157 


Ex Hachina: Coapiters 


PHYS 


102 


Intro to Pnysics u 


ECO 


122 


Prin of Micro Econ 


PHYS 


111 


Principles ox raysics 


ECOS 


121 


Prin of Hacro Econ 


PHYS 


160 


Physics ox so4xna 


EC08 


321 


Honey and Banking 


PHYS 


170 


Descriptive asxroooiny 


ECOB 


341 


Microanalysis 


POL 


102 


Hedia and Politics 


EDOC 


219 


Elements of Tutoring 


pa 


103 


0.5. & the World 


EDOC 


340 


Foundations of Education 


m 


343 


Production Kanageneot 


ENGL 


112 


Introduction to Short Story 


PSY 


210 


baucaxiGSaii rsyaxxLo^y 


fis 


203 


principles of Finance 


PSY 


311 


FnjSiOlOgXCai rfljCD 


FIB 


311 


Business Finance 


PSY 


321 


insKswl rsycnoiogy 


FKES 


201 


Intermediate French 


soc 


207 


Race « EuniciLj 


FRED 


202 


Intermediate French 


SPAYH 


203 


Intro to i^peecn ujrrvc 


GBUS 


303 


Business Law 1 


SPATR 


284 


nanuaA tOHM&uvii uiaui 


GIPPE 


63 


Golf I 


sp con 


115 


CTafllnPflff 8SXI rTvI CPB 1 VA llirriUJ i 


HEAL 


450 


Psychoactive Drugs 


THEfc 


160 


Intro to meaxre 


HEAL 


350 


Psychoactive Drugs 


TV -ft 


121 


Intro to riass neaia 


HIST 


105 


Hews of the Day 


TY~K 






HRM 


306 


Organizational Behavior 


rv ~k 




TWlftviflion Directino 


HUM 


340 


personnel BaainVstxavioo 


If R 




Public Relations 


LEW 


233 


Intro to linguistics 


TV-R 


241 


Advertising 


tu 


111 


Introduction to Business 


TV-R 


2% 


Audience Research 




310 


Quantitative Hathads in Hanageraent 


TV-R 


122 


Ken Telecom Tech 


« 


421 


Business Policy 


MSB 


108 


Foundations of Writing 


HATH 


105 


Hath for Decision Waking 




315 


Technical Writing 


SNH 


108 


Calculus for Decision flaking 








HATH 


243 


Statistics 








HKTJK 


323 


Consumer Behavior 
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Table 9 

Ithaca Courses Targeted in Faculty Interview and the CM Linkage 
to Gains in Regular Mathematics 



— 

Course 


— — --— 

Rafter 


Course Title 


Course Umber 


course Title 


MB 


205 


"*»~^--—~—-—^"^ — 

Critical Health Issues 


THE* 


160 


Intro to Tneatre 


AHS 


401 


Public Health and Epidoiiology 


TV-R 


121 


Intro to Mass Media 


MTHRO 


104 


Cultural Anthropology 


TV-R 


296 


Audience Research 


AHTHBO 


104 


Cultural anthropology 


PHYST 


103 


Introduction to Healtn rrozessiODS 


BIO 


101 


Fundamentals of Biology 


PHYST 


203 


Intro to Physical Iter 


BIO 


201 


Anatomy and Physiology 


PHIST 


311 


Physical Therapy Proc 


BIO 


202 


taatony and Physiology 


PHYST 


342 


Henroanatasy & lieuropb 


BIO 


112 


Food and Fed Control 


SPftTH 


284 


Manual Ccmmicatloos 


CHEH 


102 


Cantenporary Chaalcal Issues 


PHYS 


101 


Intro to Physics I 


CffiR 


111 


Fundamentals of Chemistry 


PHYS 


102 


Intro to Physics 11 


CHEX 


112 


Organic & Biochemistry 


PHYS 


111 


Principles of Physics 


CHEW 


113 


Chanistry Lab 


PHYSED 


125 


Hasan Anatasy 


CCH 


102 


Career Development 


PHYSED 


126 


Huaan Anatcosy lab 


CSCI 


no 


Introduction to Lata Processing 


PHYSED 


246 


Mot Available 


ECOM 


121 


Prin of Macro Icon 


PHYSED 


237 


Blcnechanlcs of Huaans 


ECOH 


122 


Prin of Micro fccon 


PHYSED 


334 


Physiology 


ECOH 


321 


Hooey and Banking 


PHYSED 


335 


Physiology 


ECC* 


341 


Microanalysis 


PHYST 


331 


Histology & Pathology 


EDflC 


219 


Elegants of Tutoring 


SP&TH 


203 


Intro to Speech Correc 


EDOC 


340 


Foundations of Ednc 


TV-R 


232 


Public Relations 


ENGL 


110 


Introduction to Fiction 


y&R 


315 


Technical Writing 


ENGL 


112 


Introduction to Short Story 




108 


Foundations of Writing 


FIN 


311 


Business Finance 








GBOS 


303 


Business Lav I 








HEAL 


350 


Psychoactive Drugs 








HEAL 


450 


Psychoactive Drugs 








WW 


250 


Labor Relations 








HSU 


306 


Organizational Bah 








LOG 


233 


Intro to Linguistics 








TO 


310 


Quant Meths in Hng&t 








n 


310 


Quant Meths in Hngmt 








n 


421 


Business Policy 








fttTH 


108 


Calculus for Decision Making 








HATH 


243 


Statistics 










323 


Consumer Behavior 








MU 


102 


Medical Terminology 








POTSED 


81 


?Ws Football 








PHYS 


170 


Descriptive istronasy 








POL 


102 


Media and Politics 








POL 


103 


D.S. & the World 








PM 


343 


Production Management 








SP COM 


115 


BnsiTWfSK PmfA«innal Coommicatinn 








SOC 


207 


Race & Ethnicity 









TaD^e 10 . , 

Ithaca Courses Targeted in Faculty Interviews and the CAM Linkage 

to Gains in Reading Comprehension 



Course 


Number 


course i iiic 


AC 


105 


Principles oi mccx, i 


AC 


105 


Principles oi accu i 


BIO 


104 


Environnven vBi isioiogy 


ECON 


121 


Prin of Macro ficon 


ECON 


122 


Prin of Micro Econ 


FIN 


203 


Principles of Fin 


GIPPE 


63 


Golf I 


GBUS 


303 


Business Law i 


HUM 


306 


Organizational Ben 


HUM 


340 


Personnel Aarainistra 


MA 


111 


Introduction to Bus 


MA 


310 


Quant Metns in wngait 


MA 


421 


Business Policy 


MATH 


105 


Math for Decision naKing 


MATH 


105 


Matn tor uecision nailing 


MKTNG 


323 


Consumer Behavior 


PHYS 


160 


Physics of Sound 


PHYSED 


125 


Human Anatomy 


PHYSED 


126 


Human Anatomy Lab 


PHYSED 


237 


Biomechanics of Humans 


PHYSED 


334 


Physiology 


PHYSED 


335 


Physiology 


TV-R 


232 


Public Relations 


TV-R 


241 


Advertising 
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Table 11 

Ithaca Corses Targeted in Faculty Interviews 
to Gains in Quantitative Comparisons 



and the CAR Linkage 



Course 


Rater 


Coarse Title 


Course Ifrnb 














BIO 


101 


Fuodanentals of Bio 


PHYSED 


246 


BIO 


201 


Rnatcoy and Physiology 


PSY 


210 


BIO 


202 


toatcB? and Physiology 


PSY 


311 


CHEN 


102 


Cootesporary Chemical Issues 


PSY 


321 


CHRH 


111 


Fundamentals of Chen 


SP&TH 


203 


CHEM 


112 


Organic & Blochenlstry 


SPITH 


284 


CHEH 


113 


Chemistry Lab 


SP COM 


115 


csa 


157 


Ex Madvina: Coopatera 


TV-R 


131 


ECCB 


321 


Honey and Banking 


TV-R 


202 


son 


341 


Kicroanalysis 


TV-R 


322 


EDOC 


219 


Elements of Tutoring 


w&r 


1QB 


EDOC 


340 


Foundations of Sduc 


WG8 


315 


DHL 


112 


Introduction to Short Story 






FEB 


311 


Business Finance 






FREK 


201 


Intermediate French 






GBOS 


303 


Business Lair I 






HEAL 


350 


Psychoactive Drugs 






HEAL 


450 


Psychoactive Drugs 






HRH 


250 


Labor Relations 






HB1 


306 


Organizational Behavior 






HIST 


105 


Hews of the Day 






LING 


233 


Intro to Linguistics 






n 


310 


Quant ffeths in ftopst 






ha 


421 


Business Policy 






MATH 


108 


Calculus for Decision Baking 






HATH 


243 


Statistics 






HRA 


102 


Radical Terminology 






HU ED 


105 


Wind Ensemble 






PHIL 


151 


Reasoning I 






PHYST 


203 


Intro to Physical Ther 






PHYST 


311 


Physical Therapy Proc 






PHYST 


331 


Histology & Pathology 






PHYST 


342 


neuroanatomy & feuropb 






PHYS 


101 


Intro to Physics I 






PHYS 


102 


Intro to Physics II 






PHYS 


111 


Principles of Physics 






PHYS 


170 


Descriptive Astronomy 






PHYSED 


125 


Human Aaatcmy 






PHYSED 


126 


Hunan taatcmy Lab 






PHYSED 


237 


Biomechanics of Humans 






PHYSED 


334 


Physiology 






PHYSED 


335 


Physiology 







Course Title 



Dot Available 

Educational Psychology 

Physiological Psychology 

Abnormal Psychology 

Intro to Speech Correc 

tenual Casomicatinns 

Business and Professional Cccnani cation 

Media Writing 

Television Directing 

Hen 'Means Tech 

Foundations of Writing 

Technical Writing 




Table 12 

Ithaca Courses Targeted in Faculty Interviews and the CAM Linkage 
to Gains in Antonyms 



Course 



Number 



Course Title 



ENGL 
FIN 
GIPPE 
PHYS 



110 
203 
63 
160 



Introduction to Fiction 
Principles of Finance 



Golf I 

Physics of Sound 



Faculty Responses 

Through the quantitative procedures and analysis , individual courses were 
identified that were associated with gains in the general learned abilities of 
students and subsequently faculty were identified who had taught these 
particular courses and were currently employed at Ithaca College, 

As Table 13 indicates, the majority of faculty (77,6%) perceived that 
their courses helped students to improve in Logical Reasoning, followed by 63,5% 
of faculty who believed their courses helped students in Analytical Reasoning 
and Reading Comprehension, and 60*7% viewed their courses as beneficial in the 
area of Sentence Completion. 
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Table 13 

Faculty Perceptions of General Learned Abilities at Ithaca College 



Sentence Reeding Regular Quantitative Data Logical taalyti 

Conpletion analogies Co^rehansion tatonyas Hatbeaatics Conparisons Interpretation Reasoning Raaaoai 

(n=*8 



By course helps 


60.75% 


57.94% 


63.55% 


55.14% 


46.73% 


24.30% 


59.81% 


77.57% 


students inprove 


(n=€5) 


lo=62) 


(n=68) 


(n=59) 


(n=50) 


(n=26) 


(n=64) 


(n=83) 


My course does 


38.32* 


41.12% 


34.58% 


47.94% 


51.40% 


75.07% 


40.19% 


22.43% 


not help 


(n=41) 


(n=44) 


(n-37) 


(n=48) 


(n=55) 


(n=81) 


(n-43) 


(n=24) 


students inprove 


















Don't know 


.93% 


.93% 


1.87% 


0 


1.87% 


0 


0 


0 




(n=l) 


(n=l) 


(n=2) 




&-2) 









The faculty responses of whether they considered their own courses as 
enhancing a student's ability to respond to each of the nine item-type areas 
were compared with the results from the quantitative analysis. Recall that 
through the Cluster Analytic Model, courses were identified and their 
associations with the nine measures of general learning. Table 14 indicates 
that there were high levels of agreement for the Reading Comprehension, 
Analytical Reasoning, and Regular Mathematics item- types. Faculty believed 
their courses helped students to improve in these areas and the quantitative 
results supported their perceptions. Overall, faculty seemed to understand what 
abilities these item- types were measuring. However, low levels of congruency 
existed between the faculty responses and the quantitative results prijnarily for 
the Antonyms and Quantitative Comparisons item-types. However, the quantitative 
analysis indicated that there were courses associated with student improvement 
in these areas. The main reason for the incongruence of results between the 
faculty perceptions and the quantitative analysis involve the actual interview 
instrument used. Faculty were given examples of the item- types taken directly 

- n- 
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from a Graduate Record Examination. They often focused on the content of the 
specific question rather than the broader goal of what ability the item-type was 
measuring. Also, these item-types explained 20 percent or less of score 
variance . 



Table 14 

Level of Agreement Between Responses from Faculty Interviews and 
Quantitative Analysis for Ithaca College 



GRE ITEM-TYPES 



Reading Regular Quantitative Analytic 

Comprehension Antonyms Mathematics Comparisons Reasoning 

Percentage of 
agreement that 

course helps to 70.73% 20.00% 48.53% 25.45% 66.13% 

improve learned 

ability 



Faculty who responded that their courses helped students improve their 
abilities in relation to the GRE item-types were asked how their individual 
courses helped. They were also asked what courses to recommend for students to 
take if they wanted to improve their abilities. Specific examples from the 
faculty examinations supplement the faculty interview responses in reference to 
the four major item-types of Analytical Reasoning, Reading Comprehension, 
Quantitative Comparisons, and Regular Mathematics. 

Analytical Reasoning 

Some faculty believed that their courses helped students to improve in 
Analytical Reasoning- A political science professor stated that through a major 
written paper in his course, students must analyze a situation such as the 
Vietnam war and make a written case of whether it was winnable. A busineb? 
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professor stated that her consumer behavior course helped students since it 

dealt with the changing environmental conditions and students drew conclusions 

of how these conditions affected certain problems. A science professor thought 

that his biology course helped students improve Analytical Reasoning through thi 

extensive number of clinical decisions that the students made. 

A test item from a philosophy professor's examination is an example of how 

Analytical Reasoning is assessed in the course. 

The following problem takes place in the Forest of Forgetf ulness . The 
Lion lies on Mondays, Tuesdays, and Wednesdays and tells the truth on 
the other days of the week. The Unicorn lies on Thursdays Fridays, 
and Saturdays and tells the truth the other days. One day Alice met 
the two in the foiest. 

The Lion said "I lied yesterday." 

The Unicorn said "I lied the day before yesterday or I will tell the 
truth tomorrow. M 

Can it be determined what day of the week it is? 

If so, what day? 

A biology professor's examination also provides a test item that involves 
Analytical Reasoning. 

From plant A the coleoptile was removed and placed on an agar block. 
The agar block was then placed on top of portion 2 of the same plant. 
Exposed to light, it bent toward the light source. 

From plant B section 1 was removed, then replaced in its original 
position. Exposed to light, it bent toward the light surface. 

In plant C, the coleoptile (1) was removed and discarded, being 
replaced by and untreated agar block. Exposed to light, nothing 
happened. 

What do these three plants demonstrate relative to the cause of the 
bending of the plant. Explain the purpose of plants A, B, and C in 
this experiment. 

Faculty perceived that their own courses helped students to gain in 
Analytic Reasoning and their examinations provided illustrations of how 
this type of learning was evaluated. 
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Reading Comprehension 



Some faculty believed that their courses improved students' Reading 
Comprehension abilities. An economics professor stated students gained an 
understanding of arguments and the cause/effect relationships through the 
readings involved in the textbook and the course content. A marketing professor 
used sets of mini-cases to apply the material learned in class through assigned 
readings. An advertising professor assigned a great deal of reading material 
and required students to read the Mew York Times. He emphasized words and the 
use of language in his course primarily through the extensive reading 
assignments. Another advertising professor stated that his course involved the 
practice of using reading comprehension skills for creating persuasive 
strategies. A math professor used word problems in his course. 

A business professor's examination in Consumer Behavior illustrates a test 
item involving Reading Comprehension. 

Tandy Corporation is attempting to reposition its consumer business by 
changing its current "low budget", "frumpy", image to one as America's 
technology company". Which element of their marketing mix is being 
changed most dramatically in order to accomplish this a^sired change 
in image? 

a . Product 

b. Price 

c . Promotion 

d. ^lace 

Another business professor's exam in an Introduction to Business course 
demonstrates a Reading Comprehension item. 

Twenty nonunion employees of the American Telephone and Telegraph 
Company (AT&T) and the Chesapeake & Potomac Telephone Company (C&P) 
filed suit against the Communication Workers of America (CWA) in 
1986. AT&T and C&P maintained labor agreements containing agency shop 
provisions with the union. In their lawsuit the nonunion employees in 
the bargaining unit objected to CWA's use of their agency fees for the 
purposes unrelated to collective bargaining, contract administration, 
or grievance adjustments. The judge found CWA guilty of misusing 
agency shop dues. In her findings the judge should have indicated 
that CWA action violated the 
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a. Taft-Hartley Act 

b. Norris-LaGuardia Act 

c. Wanger Act 

d. Ladrum-Griffin Act 

e. Fair Labor Standards Act 

Faculty viewed their own courses as aiding students in Reading Comprehension. 
The test items reveal that faculty did evaluate the student' t ability to 
comprehend narrative and understand the implications. 

Quantitative Comparisons 

An advertising professor stated that in his audience research course 
students were given sets of information and made comparisons for decisions. An 
anatomy professor's course dealt with work in math and examined relationships in 
data. A business professor stated that correlation, regression, and advanced 
math techniques were used in her course. A chemistry professor to a limited 
extent covered basic algebra. 

A biology professor's examination demonstrates a test item similar to the 
GRE item-type of Quantitative Comparisons. 

If all else is equal, which subject will have the highest rate of blood 
flow? 

Subject A Subject B 

B.P. = 130/80 120/80 
X AP = 102 90 
Central Venous Pressure 2 

A. subject A 

B. subject B 

C. both subjects will be equal 

A physics professor's test item also illustrates a similarity to the 
Quantitative Comparison item-type. 

You have two ton' 90 dB at 123 Hz and 90 dB at 1500 Hz. Is one 
louder than the c. lier and if so by how many times? Show your work! 




Both of these test items require students to make comparisons of two 
quantities. These examples illustrate how faculty evaluate this type of 
learning. 



Regular Mathematics 

Some faculty believed their courses improved students' abilities in Regular 
Mathematics- An economics professor stated that his course used math formulas 
to address theories and applied work problems. A professor of business policy 
stated that his course used spreadsheets and balance sheets in order to solve 
business operation problems. A biology professor required a major course 
project involving diet analysis. 

A physical therapy professor's examination in neuroanatomy/ neurophysiology 
provides an illustration of an item similar to the Regular Mathematics 
it em- types. 

A neuron has a resting membrane potential of -80 millivolts, and an 
excitatory threshold of -60 millivolts. The dendritic spines of this 
neuron receive input from the pre- synaptic terminals of about 50 other 
neurons. Each pre-synaptic terminal is capable of producing an EPSP 
of 3 millivolts. What is the minimum number of pre-synaptic terminals 
which must fire in order to cause an action potential in the neuron? 

a. 3 a, 21 

b. 5 e. all of them 

o. 7 

A business professor's final examination for a Quantitative Methods in 
Management course demonstrates another example of similarity with Regular 
Mathematics. 

Total yearly sales achieved by the members of your sales force are 
normally distributed. The mean is $1,000,000 and the standard 
deviation is $2,000,000. Joe's yearly sales are $1,400,000, The most 
accurate statement about his position is: 

a. in the top half, but not in the top quarter 

b. in the top quarter, but not in the top ten percent 

c. in the top ten percent, but not in the top five percent 

d. in the top five percent 

46 
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These examples involve arithmetic and ask students to perform mathematical 
calculations. 

Implications for Academic Advising at Ithaca College 

Faculty were asked to recommend courses that would help students answer 
sample questions taken from the GEE examination used in this study. 
Faculty thought that courses in mathematics, science, logic, economics, and 
philosophy helped students improve their abilities in Analytical Reasoning. 
Some faculty recommended these general areas while other faculty recommended 
specific courses offered at Ithaca College such as Math for Decision Making. 
However, the cluster analysis indicated that other specific courses in health 
care, history, political science, and advertising were associated with student 
improvement in Analytical Reasoning. 

Most faculty thought that English, literature and writing courses helped 
students improve Reading Comprehension. However, other specific courses in 
music, mathematics, accounting, finance, and marketing were associated with 
student improvement in Reading Comprehension. 

Faculty thought that math, computer science, science courses such as 
biology, chemistry, statistics, social science research and economics helped 
students to improve their abilities in Quantitative Comparisons. Most faculty 
recommended these general courses while some faculty recommended specific 
courses such as Math for Decision Making or Microeconomics. 

Faculty thought that math, physics, chemistry, economics, and statistics 
helped students to .rprove their abilities in Regular Mathematics. However, 
other specific courses in Physical Therapy, Speech Pathology, Psychology, and 
Accounting were also associated with Regular Mathematics as determined by the 
quantitative analysis. 
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Faculty clearly had difficulty making specific course recommendations 
beyond their immediate discipline or field of expertise- Instead, they would 
roost frequently identify a discipline, field of study or department that they 
believed would be able to help the student. Faculty drew upon their 
undergraduate and graduate experience, the Ithaca College catalog (Undergraduate 
Announcements ), and word of mouth information from other faculty and former 
students to make specific course recommendations. When faculty did not know how 
to advise the student, their course reconaiendations were understandably vague 
and general* When faculty saw a similarity between the abilities tested in a 
GRE questionnaire and that which they tried to develop in their class, they were 
able to quite articulately explain how they tried to encourage student 
development - 

Mills College Samples and Dominant Item-Types 
A cluster analysis of coursework and assessment scores for the Mills 
samples produced a list of courses where students had inyroved on one or more of 
the GRE nine item- types. The faculty who taught these courses were identified. 
From this group, 34 faculty agreed to be interviewed individually to assess 
their perception of their courses in relation to the general learned abilities 
of college students. Some faculty interviewed taught more than one course 
associated with improvement in student learning and responses were obtained for 
both courses- 

For Mills Sample #1, four item- types were dominant and associated with 
student improvement . Seventeen percent of the variation in Sample #1 test 
scores was explained by the Quantitative Comparison item-type. Another ten 
percent was explained by Antonyms. Six percent of the variation in Sample #1 
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was explained by Analytic Reasoning and Data Interpretation (Ratcliff, 1990d). 

For Sample #2, forty percent of the variation in their test scores was 
explained by Analytical Reasoning and another twenty-five percent was explained 
by the Reading Comprehension item- type. Eighteen percent was explained by 
Quantitative Comparisons and Regular Mathematics. Eight percent was explained 
by the Antonyms and Logical Reasoning item- types (Ratcliff, 1990d) . 

The following discussion represents an aggregation of the data across these 
two samples since improvements in student learning were associated with some 
similar item- types. Tables 15 through 19 indicate the Mills College courses 
represented in the faculty interviews and their association with the 
item-types. These relationships were identified from the quantitative 
procedures utilized in the Cluster Analytic Model. 



Table 15 

Courses Represented in Faculty Interviews at Mills College Associated 
with Improvement in Quantitative Comparisons 



Course Number Title 



ANTH 


58 


Cultural Anthropology 


ARTH 


186 


Japanese Painting & Prints 


BIO 


42 


Human Physiology 


DANCE 


5 


Elementary Modern Dance Techniques 


DANCE 


5 


Elementary Modern Dance Techniques 


DRA 


63 


Maxe-Up 


EDUC 


101 


Social Foundations of Education 


ENG 


88 


Conanunication Aesthetics & Criticism 


ENG 


56 


Writing of Fiction & Verse 


FRENCH 


10 


Composition & Text Analysis 


GOVT 


102 


Administrative Behavior 


HIST 


139 


Diplomatic History of U.S. 


MUS 


27 


Chamber Music 
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Table 16 

Courses Represented in Faculty Interviews at Mills College Associated 
with Improvement in Analytical Reasoning 



Course 


Number 


Wl 11,1 

Title 


ATRH 


18 


Introduction to Western Art 


CHEM 


4 


Introduction to College diemistry 


DANCE 


15 


Jazz Dance 


ECON 


135 


Microeconomic Theory 


ECON 


81 


Introduction to Statistics 


ENG 


Gl 


Genre Courses 


FRENCH 


40 


French Pheonetics 


GOVT 


90 


American Foreign Policy 


HUS 


27 


Chamber Music 


SOC 


99 


Criminology 


soc 


142 


Medical Sociology 


SOC 


61 


Women in Contemporary Society 


soc 


103 


Women & Work 



Table 17 

Courses Represented in Faculty Interviews at Mills College Associated 
with Improvement in Data Interpretation 



Course 


Number 


Title 


ANTH 


58 


Cultural Anthropology 


DRA 


63 


Make-Up 


EDUC 


101 


Social Foundations of Education 


ENG 


10 


Introductory Seminar in Writing about Literature 


ENG 


88 


Communication Aesthetics & Criticisms 


ENG 


10 


Introductory Seminar in Writing about Literature 


FRENCH 


57 


Composition & Text Analysis 


GOVT 


17 


International Relations 


HIST 


139 


Diplomatic History of U.S. 


MUS 


107 


Individual Instrument Instruction—Piano 
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Table 18 

Courses Represented in Faculty Interviews at Mills College Associated 
with Improvement in Reading Comprehension 



Course 


Number 


f 4 4-1 £% 
X XLXB 


mi nn 


18 


Introduction to Western Art 


ECON 


81 


Introduction to Statistics 


ENG 


61 


Genre Courses 


FRENCH 


40 


French Pheonetics 


FRENCH 


2 


Elementary French 


FRENCH 


3 


Intermediate French 


FRENCH 


4 


Intermediate French 


HIST 


120 


Ireland: Culture & Conflict 


HIST 


11 


History of Western Thought 


PSCH 


49 


Fundamentals of Psychology 









Table 19 

Courses Represented in Faculty Interviews at Mills College Associated 
with Improvement in Antonyms 



Course 


Number 


Title 


ANTH 


161 


Cross-Cultural Perspective: Women 


ARTH 


186 


Japanese Painting & Prints 


GOVT 


148 


Model United Nations 


M&CS 


4 


Discrete Mathematics 


M&CS 


151 


Modern Algebra 


M&CS 


8 


Linear Algebra 


SOC 


55 


Introduction to Sociology 


SOC 


94 


Sociology of Mass Communication 



Faculty Responses 

Many faculty at Mills believed their courses helped students gain in the 
general learned abilities. As Table 20 indicates, the majority of faculty 
(85.7%) perceived that their courses helped students to improve in Reading 
Comprehension, followed by 78.6% of faculty who believed their courses helped 
students in Logical Reasoning, and 64.2% viewed their courses as beneficial in 
the areas of Analogies and Analytic Reasoning. 
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Table 20 

Faculty Perception of General Learned Abilities at Mills College 



Sentence Seeding Regular Quantitative Data Logical Analytical 

Ccopletico Analogies Ccqpw^aasian Antcnytas Hathcwrtics Comparisons Interpirtatioo Reasoning Beaming 



My course helps 
students iaprore 

My course does 
cot help 
students improve 

Doc 1 ! know 



57.14% 


64.29% 


85.71% 


54.76% 


26.19% 


21.43% 


40.48% 


78.57% 


64.29% 


(n=24) 


(o=27) 


(ii=36) 


(d=23) 


(n=ll) 


(n=9) 


(n=17) 


(n--33) 


(n=27) 


42.86% 


35.71% 


14.29% 


45.24% 


71.43% 


78.57% 


59.52% 


21.43% 


30.95% 


(n=18) 


(n=15) 


(o=6) 


(o=19) 


(n=30) 


(n=33) 


(d=25) 


(n=9) 


(n=13) 



.00% 
(0=0) 



.00% 
(n=0) 



.00% 

(o=0) 



.00% 

(0=0) 



2.38% 

(n=D 



.00% 

(D=0) 



.00% 
(o=0) 



.00% 

(n=0) 



4.76% 
(o=2) 



The faculty responses of whether they considered their own courses as 
enhancing a student's ability to respond to each of the nine item-type areas 
were compared with the results from the quantitative analysis. Recall that 
through the Cluster Analytic Model, courses were identified and their 
associations with the nine measures of general learning. Table 21 indicates 
that low levels of congruency existed between the faculty responses and the 
quantitative results primarily for the Data Interpretation, Antonyms, and 
Quantitative Comparisons item-types. Two reasons account for some of these 
differing viewpoints. The first reason for the incongruence of results between 
the faculty perceptions and the quantitative analysis involve the actual 
interview instrument used. Faculty were given examples of the item-< ypes taken 
directly from a Graduate Record Examination. They often based their remarks on 
the content of the specific question rather than the broader goal of what 
ability the item-type was measuring. For example, in the area of Data 
Interpretation, faculty were given a test item with a graph of data pertaining 
to a sample. They viewed this question as dealing mostly with similar graphs 
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and were unable to see connections with their own courses. However, Data 
Interpretation items were intended to measure the student's ability to make 
decisions about "data presented in tables or graphs and test one's ability to 
synthesize information, to select appropriate data for answering a question or 
to determine that sufficient information for answering a question is not 
provided (ETS, 1989, p. 39). A second reason for the lack of congruence is that 
these item-types did not explain the majority of score variance among students 
in either sample. 

There were higher levels of agreement for the Reading Comprehension and 
Analytical Reasoning item-types. More faculty believed their courses helped 
students to improve in these areas and the quantitative results supported their 
perceptions. Overall, faculty seemed to better understand what abilities these 
item- types were measuring. 



Table 21 

Level of Agreement Between Responses from Faculty Interviews and 
Quantitative Analysis for Mills College 

GRE ITEM-TYPES 

Data Reading Antonyms Quantitative Analytic 

Interpretation Comprehension Comparisons Reasoning 



Percentage of 
agreement that 

course helps to 30.00% 90.00% 37.50% 23.08% 69.23% 

improve learned 

ability 



Faculty at Mills College who responded that their courses helped students 
improve their abilities in relation to the GRE item-types were asked how their 
individual courses helped. They were also asked what courses to recommend for 
students to take if they wanted to improve their abilities. Specific examples 



from the faculty examinations supplement the faculty interview responses in 
reference to the five major item-types of Analytical Reasoning, Quantitative 
Comparisons, Antonyms, Reading Comprehenion, and Data Interpretation - 

Analytical Reasoning 

Some faculty believed their courses helped students to improve in 
Analytical Reasoning. An art history professor gives students a tray of slides 
to study over the duration of the course and then they are given new slides to 
identify and analyze based upon the concepts they have learned. This 
professor's examination in Western Art provides a test item illustrating the 
necessity of analytical skills. 

This is a portal on the south side of a medieval European church. 
Analyze this example and classify it according to the characteristic 
of Romanesque or Gothic art. 

A sociology professor requires her students to do analysis in class. For 

example, students analyze gender roles as outsiders observing the inner 

structures of a family. Her exam contains an essay question which deals with 

role stereotypes. 

Explain de Beauvoir's self -other dichotomy. Include in your 
discussion the concept of "identification with the oppressor" 
(aggressor). How do these conceptual tools explain the persistence of 
gender role stereotypes? 

Faculty examinations indicated that a student's ability to analyze were 
assessed. 

Quantitative Comparisons 

Some faculty believed their courses helped students to improve in 
Quantitative Comparisons. An economics professor stated that students in her 
course completed maximization problems to determine which quantity is larger. A 
statistics professor has r-tudents comparing their results to other quantitative 

5 4 

- 46 - 



entities to determine whether one result is greater or less than another. 
Similar with the findings at Stanford University and Ithaca College, faculty at 
Mills were unable to see clear linkages with their courses helping in 
Quantitative Comparisons. One reason for this difficulty was the actual item in 
the interview instrument was a geometric figure and faculty tended to focus on 
this visual image rather than the broader abilities associated with Quantitative 
Coim?arisons . 

Second, the majority of courses associated with gains in Quantitative 
Comparisons were humanities and creative arts courses rather than quantitative 
or mathematical courses. None of the exams collected for courses associated 
with improvement in Quantitative Comparisons contained test items similar to 
this item- type. 
Data Interpretation 

Some faculty believed their courses helped students to improve in Data 
Interpretation. An anthropology professor stated that her students obtain a 
great deal of practice in interpreting tabular data. A chemistry professor 
emphasized that data interpretation is the foundation of his course. Students 
used observational data which was frequently experimental and interpreted the 
results. 

A cultural anthropology exam contains a test item which involves the 
interpretation of data relevant to the discipline. 
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Assume this is a chart for a matr ilineal , matrilocal society. Answer 
the following, using numbers where appropriate. 



B. 

C. 



Shade in all the members of EGO's clan. 
Who is the head of EGO's clan? 

If this society prefers matrilateral cross-cousin marriage, who 
should EGO marry? 

D. Who will EGO live with after he is married? 

E. If EGO has a son, who will have authority over him (the son)? 

In general, though, the majority of faculty believed their courses did not help 
students to develop data interpretation skills. It was difficult to find 
examples of Data Interpretation in the faculty examinations especially since 
many humanities and creative arts courses were associated with improvement in 
Data Interpretation. 



5'* 



9 

ERIC 



- 48 - 



antonyms 

Some faculty believed their courses helped students to improve in 
Antonyms, A professor who teaches criminology stated that students in his 
course define and discuss opposing concepts such as criminal versus non-criminal 
activities or morality versus no morality. In a similar manner, a history 
professor stated that his course aids students in Antonyms since he teaches the 
precision of language through an examination of historical documents. 

A sociology of mass communications course test item provides an example of 

Antonyms where students note the differences between two types of programming. 

What are the major differences between day time soap operas and prime 
time prograiraning? Focus on three major areas of contrast , and 
describe the differences between "soaps" and "prime time." Finally, 
for each "difference" that you have identified, give a brief (one or 
two sentence) explanation of "why" this difference is significant. 

Approximately one-half of the faculty at Mills thought thei: courses helped 

students to improve in Antonyms. However, it was difficult to find exanples of 

this area in the faculty examinations. 

Reading Comprehension 

Some faculty believed their courses helped students to improve in Reading 
Comprehension. An English professor teaches her students to read the text 
carefully in her course. They learn to identify the main ideas and supporting 
statements by searching for clues in the text. Students also learn to 
differentiate between statements and to discover their implications. An art 
history professor emphasized reading for her students so that they understand 
both the content and context of art. 

A professor's exam in the fundamentals of psychology illustrates a test 
item involving Reading Comprehension. 
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Mary takes a course in which she is tested every two weeks. Her 
studying falls off right after a test, followed by a gradual increase 
to a rapid rate of studying as the next test approaches. Her studying 

conforms to the typical pattern of responding maintained on 

schedules . 

A. fixed- ratio 

B. variable-ratio 

C. fixed- interval 

D. variable- interval 

Humanities courses were frequently associated with improvement in Reading 
Comprehension. A review of the syllabi in these particular courses revaaled an 
emphasis on required readings often supplemented with a series of discussion 
questions. However, many of the examinations in these courses did not test 
Reading Comprehension directly. Instead, the examinations often focuaed on 
testing the analytical abilities of students. 

Implications for Academic Advising a t Mills College 

Fp^ulty were asked what courses to recommend for students to take if they 
wanted to improve their abilities as defined by the item-type areas of the GRE. 
In general, faculty did not have a consistent knowledge base for making such 
decisions. They drew upon their own undergraduate experience (usually at 
another college or university), The Mills C ollege Bulletin, word of mouth 
recommendations of courses by other faculty or students. Faculty thought that 
courses in logic, statistics, research methods, philosophy, and theory-related 
courses helped students improve their abilities in Analytical Reasoning. Some 
faculty recommended these general areas while other faculty recommended specific 
copies offered at Mills College which were usually courses in their own 
disciplines. Faculty were less knowledgeable about recommending courses outside 
of their disciplines. Since the majority of faculty were associated with 
liberal arts-related areas, these types of courses were frequently recommended. 
However, the cluster analysis indicated that other specific courses at Mills in 
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chemistry, economics, and music were associated with student improvement in 

Analytical Reasoning as well. 

Faculty thought that courses in mathematics, geometry, and logic helped 
students improve their abilities in Quantitative Comparisons. Again, faculty 
recommended specific courses offered at Mills within their own disciplines or 
general traditional subject areas. However, the cluster analysis indicated that 
other spc ific courses in biology, dance, art history, and music were associated 
with studant improvement in Quantitative Comparisons. 

Faculty thought fiat courses in mathematics, economics, social science 
methods, and math helped students improve their abilities in Data 
Interpretation. However, the cluster analysis indicated that other specific 
courses in French, English, anthropoid/, and music were associated with student 
improvement . 

Faculty thought that courses in English, public speaking, literature, 
history, and foreign language would help students improve their abilities in 
Antonyms. Most faculty recommended these general areas. However, the cluster 
analysis indicated that other specific courses in mathematics (algebra- related 
courses), anthropology, and sociology courses were associated with student 
improvement. This research demonstrated that there were many different types of 
courses and disciplines associated with improvement in student learning. 
Faculty were tentative in recommending general traditional subject areas. When 
faculty viewed direct linkages of their own discipline with a certain item-type, 
they were comfortable in suggesting specific courses in their own discipline. 
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FINDINGS OP DCP FACULTY SURVEY: PART TWO 

Introduction 

As noted earlier, Part Two of this report examines faculty perceptions of 
the courses they teach. More specifically, faculty members from Stanford, 
Ithaca, and Mills were asked about the nature, purpose, and form of student 
evaluation, the desired outcomes of cl:.ss instruction, course planning 
strategies, and class scheduling. What follows, then, are current faculty 
perspectives regarding college students' educational experiences. 

The work of several curriculum and learning theorists provided the basis 
for the development of many of the questions in the survey concerning faculty 
instructional goals, assessments, and teaching methods. For example, the 
survey's first four questions, which focused on the evaluation of student 
learning, were constructed using Bloom's 11956) cognitive domain taxonomy, 
Krathwohl's (1964) affective domain taxonomy, and Simpson's (1972) psychomotor 
domain taxonomy. In this context, a domain simply refers to a specific category 
or arena of educational development. Similarly, a taxonomy, as it is used here, 
denotes a classification of various goals of the educational process that is 
intended to assist educators with the design and evaluation of course curricula. 

A brief description of each of the domains above may be instructive. The 
cognitive domain describes educational outcomes that require "the recall or 
recognition of knowledge and the development of intellectual abilities and 
skills (Bloom, 1956, p. 7). An example of an educational outcome within the 
cognitive domain is a student's ability to remember dates, events, persons, or 
similar bits of information. 

The affective domain consists of educational objectives that are linked 
with a student's "interests, attitudes, appreciations, values, and emotional 
sets or biases" (Krathwohl, 1964, p. 7). An individual's active participation 




iu c±diss discussions is an example of an educational outcome within the 
effective domain. 

Finally, the psychomotor domain is comprised of educational goals that 
focus on "some muscular or motor skill, some manipulation of material and 
objects, or some act which requires a neuromuscular coordination" (Krathwohl, 
1964, p. 7). For instance, an educational outcome that characterizes the 
psychomotor domain is a student's ability to conduct a physical procedure with 
consistency. 

The Nature of Student Evaluation 

Faculty members were asked to indicate the degree to which the final 
evaluation of their students is based on each of the three primary areas 
discussed above: cognitive, affective, or psychomotor performances. The 
majority of faculty respondents (68%) across the three institutions in the study 
based 80 percent or more of students' final evaluation on the students 1 
cognitive development (see Table 22). Among the institutions, Stanford has the 
largest percentage of faculty (76%) who based 80 percent or more of students 1 
final evaluation on their cognitive development, followed by Ithaca (65%) and 
Mills (56%). Although marked differences exist among the institutions (owed, at 
least in part, to differences in inst : tutional priorities), a majority of those 
interviewed at each institution consider knowledge acquisition and the ability 
to use it vital to the educational process. Most faculty members perceive 
students' cognitive development to be the legitimate basis for test construction 
and course evaluation. 

The proportion of faculty members' evaluation of students 1 affective 
development is small (see Table 23). Only 14 percent of the respondents stated 
that they include students 1 affective performance as more than 30 percent of the 
final grade. Most of these individuals teach in either traditional liberal arts 




disciplines (e.g., English, Economics, History, Chemistry, Sociology, 
Literature) or applied fields (Business, Government, Engineering). 
A^oroximately one-quarter (28%) of the faculty use students' affective 
performance as 10 to 20 percent of the final grade. This corresponds with a 
finding from the syllabi analysis, m which 26 percent of the syllabi sample 
that we examined included class attendance/participation as 10 to 20 percent of 
students' final evaluation. The largest percentage (45%) of the faculty, 
representing a wide array of courses, remarked that students' affective 
development constitutes less than ten percent of the final evaluation. 

Even more pronounced is the percentage of students' final evaluation that 
is based on psychomotor competencies (see Table 24). A large majority (82%) of 
the faculty observed that students' psychomotor development is not a criterion 
for final evaluation. This finding is not unusual given that the development of 
psychomotor abilities would not be expected in the traditional courses that 
comprise a large portion of the sample. Only 42 faculty members (18%) included 
psychomotor performance as a basis for final evaluation, and most of these 
individuals (28) considered student's psychomotor development as 20 percent or 
less of a student's final grade. Faculty members who placed greater emphasis 
(i.e., 50% or higher) on psychomotor competencies in determining students' final 
grades were members of performance disciplines such as music, drama, or dance. 
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Faculty members also were asked to provide a more specific description of 
their evaluation of students' cognitive, affective, and psychomotor development 
by rating the importance that tiey attach in their overall evaluation of 
selected student abilities (using a Likert-type scale) „ A majority of 
interviewed faculty members (64% and 71%, respectively) consider the evaluation 
of students to be very important in two particular cognitive development areas: 
(1) gaining basic knowledge , language, or terms, and (2) understanding concepts, 
theories, and trends of the field of study (see Table 25). The area of 
cognitive competence ranked least important in faculty members' evaluation of 
students is students' ability to judge the worth or value of something based on 
specific criteria; students' ability to distinguish between facts and 
inferences; and students' ability to perform, act out, and demonstrate skills 
involved in the field of study. These findings suggest that students' final 
evaluations are based on the development of lower order cognitive abilities 
(e.g., knowledge, comprehension), rather than the development of higher order 
cognitive abilities (e.g., application, analysis, synthesis, and evaluation) 
(Bloom, 1972). 

The analysis of san$>le syllabi and tests, at first glance, seem to 
contradict these findings. That is, a majority (54%) of the sanple syllabi 
portray higher order cognitive skills (analysis, synthesis, evaluation) as 
expected course outcomes. Further, an even larger majority (74%) of sample 
examinations test students' higher order cognitive development (analysis, 
synthesis, evaluation). On closer inspection, however, the findings from the 
sample syllabi and tests are inflated. Each rater included highor order skill 
categories as present (usually implicitly) if a simple phrase was included in a 
syllabus or a test contained even one question that required higher order 
cognitive skill. As a result, the analysis of sanple syllabi and tests is 
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skewed upward, giving the impression that the development of higher order skills 
typically is expected from students (based on sample syllabi) and that higher 
order skills typically are necessary to complete course examinations (based on 
sample tests) . 

As noted earlier, most faculty members do not base a large portion of tneir 
final evaluation of students on students' affective development. This finding 
is reflected in faculty members' responses regarding their evaluation of 
specific affective abilities in students (see Table 26). Each component of 
students' affective development, particularly the two that focused on students' 
class participation, was thought to be "very important" or "important" by a 
considerable percentage of faculty members (ranging between 34% and 48%). Mills 
faculty ranked each affective component considerably higher than did Stanford or 
Ithaca faculty. An equally significant percentage of faculty members (ranging 
between 43% and 55%), largely from Stanford and Ithaca, rated the selected areas 
of affective development to be "unimportant," "very unimportant," or "neutral" 
in their overall evaluation of students. Faculty members who consider the 
development of students' affective abilities as an important component of their 
evaluation are scattered across many disciplines and fields. 

Students' psychomotor development played even less of a role in students' 
overall evaluation according to faculty members (see Table 27). As a result, a 
majority of interviewees (ranging between 71% and 73%) explained that each of 
the seven psychomotor development areas listed in the survey question is "very 
unimportant" in their final evaluation of students. Only those faculty members 
who teach in fields such as physical education, music education, drama, dance, 
engineering, and chemistry- -subjects that typically require some type of 
physical proficiency-considered the development of psychomotor skills important 




to students 1 learning processes. 



Table 25 

Students* Cognitive Development: Evaluation of Students' Ability to: 

Very Ver Y 
Institution Important Important Neuv.ral Unimportant Unimportant 

• Gain basic knowledge, language, or terms 

Stanford 40 37 13 1 1 

Ithaca 79 19 8 0 0 

Mills 33 6 2 0 0 

TOTAL 152 62 23 1 1 

• Understand concepts, theories, and trends of field 

Stanford 75 14 2 0 1 

Ithaca 68 30 8 0 0 

Mills 26 12 2 0 1 

TOTAL 169 56 12 0 2 

• Perform, act out, demonstrate skills involved in field 

Stanford 32 30 11 11 9 

Ithaca 38 23 22 10 13 

Mills 18 13 6 1 0 

TOTAL 88 66 39 22 21 

• Distinguish between facts and inferences; etc. 

Stanford 32 21 22 12 5 

Ithaca 38 23 22 10 13 

Mills 24 7 1 6 3 

TOTAL 94 51 45 28 21 

• Integrate learning from different areas into original idea 

Stanford 33 28 21 5 5 

Ithaca 43 31 23 5 4 

Mills 19 9 3 5 0 

TOTAL 95 68 52 15 9 

• Judge worth, value of something based on specific criteria 

Stanford 22 32 22 10 6 

Ithaca 31 31 26 9 8 

Mills 11 16 8 4 2 

TOTAL 64 79 56 23 16 

n 239 



Table 26 

Students* Affective Development: Evaluation of Students' Ability to: 



Very Ver Y 
Institution Important Important Neutral Unimportant Unimportant 

• Willingness to participate in class 

Stanford 6 21 12 12 29 

Ithaca 24 34 18 10 16 

Mills 16 13 4 2 0 

TOTAL 46 68 34 24 45 

• Interest shown through class participation 

Stanford 10 14 17 10 28 

Ithaca 22 29 26 12 13 

Mills 14 17 2 2 0 

TOTAL 46 60 45 24 41 

• Attitude toward or appreciation of class learning 

Stanford 8 19 16 11 25 

Ithaca 23 26 27 11 15 

Mills 9 14 9 2 1 

TOTAL 40 59 52 24 

• Development of consistent value system 

Stanford 8 13 19 12 27 

Ithaca 18 25 31 8 20 

Mills 8 10 7 1 6 

TOTAL 34 48 51 21 53 

n = 239 
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Table 27 

Students* Psychomotor Development: Evaluation of Students' Ability to: 

Very Very 

Institution Important Important Neutral unimportant Unimportant 

• Isolate components of physical action in performing skill 

Stanford 0 1 5 4 54 

Ithaca 5 10 12 8 66 

MiUs 2 4 0 0 22 

TOTAL 1 15 n 12 142 



• Express proper sequence for actions in physical procedures 
Stanford 1 4 1 5 54 
Ithaca 7 9 16 5 64 
Mills 2 4 0 0 22 
TOTAL 10 n _ri_ _ 10 = _ = _^°„ 

• Imitate a physical process 

Stanford 2 4 2 3 54 

Ithaca 8 8 17 3 65 

Mills 3 2 2 0 21 

TOTAL 13 H 21 0 140 

• Perform physical process with consistency 

Stanford 1 2 7 1 54 

Ithaca 13 4 14 4 66 

Mills 3 2 1 1 21 

TOTAL 17 8 22 6 141 

• Execute physical procedures 

Stanford 1 2 4 4 54 

Ithaca 8 7 17 3 66 

Mills 2 3 1 1 21 

TOTAL 11 12 22 8 141 

• Modify physical process in order to adapt to circumstances 
Stanford l 4 1 5 54 
Ithaca 6 13 15 5 62 
Mills 4 1 0 2 21 
TOTAL 11 18 16_ _ 12_ J^J_ 

m Create new or original physical procedures to fit situations 

Stanford 2 3 3 2 55 

Ithaca 2 8 18 9 64 

Mills 4 2 0 2 20 

TOTAL 8 13 21 13 139 

n = 239 
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Teaching Modes and Purposes 

Two questions in the survey were designed with reference to Axelrod's 
(1973) model of didactic and evocative modes of teaching. The didactic mode of 
teaching, characterized by the "transmitting 'knowledge 1 11 side of the continuum 
in the survey question, has as its goal a mastery of a definite body of 
knowledge. The emphasis in this model is on the acquisition of knowledge or 
skills that are obtained primarily through memorization; a teacher's goal is "to 
develop in the student an automatic or semi-automatic response 11 (Axelrod, 1973, 
P« 11). 

In contrast, the evocative mode of teaching, symbolized by the "teaching 
'thinking processes'" end of the continuum in the survey question (#5), is 
concerned with inquiry and discovery strategies. From this perspective, the 
teacher may be thought of as an artist who creates an atmosphere for learning by 
promoting students' encounters with each component of the teaching-learning 
process: teacher, learner, and subject matter. 

Most of the faculty who were interviewed believed that "teaching 'thinking 
processes'" best represented their teaching approach; that is, more than half 
(57%) of the respondents ranked their teaching approach as a "7" or above (see 
Table 28), The second single largest number (24%) of faculty stated that their 
teaching style was a balance between "transmitting 'knowledge'" and "teaching 
'thinking processes'" ("5" on the scale). Many of the faculty in this group 
objected to a distinction between these two "poles," calling the distinction 
arbitrary and stating that learning involved both content acquisition and 
development thinking processes. 

Faculty members also commented on their primary purposes in teaching (see 
Table 29). As the results indicate, the most primary objective of faculty 
members by far (37%) is to "have students learn course content." This finding 
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corresponds closely with faculty members 1 strong emphasis in their final 
evaluations on students 1 cognitive development. It also should be noted that a 
majority of Mills faculty (61%) ranked as first or second the "having students 
learn a particular perspective on course content" objective. This finding is 
not unusual given Mills' stated mission as a single-gender institution. That 
is, as a women's college, the focus of all Mills' programs, policies, and 
practices is the development of women. 

In addition, faculty informants (53%) also explained that "teaching 
students how to learn" is an important goal of teaching (i.e., ranked this 
objective either first or second). This finding relates to the previous 
question in which a majority of faculty members see themselves as "teaching 
'thinking processes. 1 " The purpose of teaching that faculty members (35%) 
considered to be furthest from primary is "assisting students to incorporate 
certain skills and/or knowledge into their daily, personal )i^es." Reflected in 
this finding is the idea, mentioned earlier, that faculty members are more 
concerned with the development of lower order cognitive skills rather than 
higher order skills such as intr ^ration, application, and evaluation. 

The findings on faculty members' evaluation of students' cognitive 
development and the findings on the nature of faculty members 1 purposes in 
teaching, when compared, present a confused picture. On the one hand, we saw 
that faculty members rated students' ability to "gain basic knowledge, language 
or terms" and to "understand concepts, theories, and trends of the field of , 
study" as very important components of their final evaluation (Question #2). 
Further, we observed that the largest number of those interviewed believe that a 
primary purpose of teaching is "having students learn course content" (Question 
#6). Finally, a large number of informants (47%) commented that "teaching 
students how tc learn" is not one of their primary teaching purposes. Each of 



these findings is most characteristic of a didactic teaching- learning process. 

On the other hand, however, a majority of faculty members (57%) stated that 
their teaching approach may be identified as "teaching 'thinking processes 1 M 
(Question #5). This finding corresponds with an evocative teaching- learning 
mode. Thus, it appears that many faculty members see themselves as evocative 
"artists" although their course objectives and evaluation procedures are clearly 
didactic. Thus, for many, content and pedagogy may not intersect in the 
classroom. Simply stated, many faculty members may be unable "to transform the 
content knowledge he or she possesses into forms that are pedagogically 
powerful" (Shulman, 1987, p. 15), 
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Table 29 

Faculty Ranking of Their Primary Purposes in Teaching 



Institution 


Closest to 
Purpose 
(Rank 1) 


Three 
(Rank 2) 


Two 
(Rank 3) 


r Ul LIlcS L X. x. VJAU 

Purpose 
(Rank 4) 


• Have students 


learn course content 








Stanford 


38 
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16 


12 


Ithaca 


43 


31 


20 


12 


Mills 
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15 


13 
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• Teach students 


how to learn 








Stanford 
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11 
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26 
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14 


5 


16 


TOTAL 


50 


74 


53 


53 


• Assist students to incorporate certain skills/knowledge in personal lives 

Stanford 5 15 21 46 
Ithaca 26 24 29 27 
Mills 12 12 9 8 

TOTAL 43 51 59 81 



n - 234 



Course Characteristics 

Most of the remaining questions in the first section of the survey were 
developed following Bergquist's (1981) suggestions for curricular implementation 
and evaluation. Accordingly, various curriculum- related issues that may have an 
impact on undergraduate learning were explored, such as when courses are 
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offered, characteristics of teachers, instructional techniques, and assessment 

strategies . 
Course Scheduling 

According to the participants in the survey, course scheduling follows what 
Bergquist (1981) refers to as traditional patterns (see Table 30). That is, 
standard semester (Ithaca and Mills) and quarter (Stanford) systems are used in 
which the vast majority of courses are taught on a yearly basis, in daylight, 
during the week, for 3-5 hours. Very few courses are offered in the summer, on 
weekends, or in the evening. 
Faculty Characteristics 

Table 31 provides a picture of faculty characteristics. As the chart 
indicates, faculty rank is fairly evenly distributed among full, associate, and 
assistant designations. In addition, a large majority of those interviewed are 
full-time faculty members (94%) whose primary responsibilities are as faculty 
(98%), rather than as administrators or departmental chairs, or mentors. As a 
result, the faculty participating in the survey are more likely to contribute to 
the overall institutional and educational environment (National Institute of 
Education, 1984). 
Who Teaches 

We asked respondents to state who typically is involved in teaching the 
course (see Table 32). As might be expected, in most cases, instruction is 
performed by an individual faculty member (86%). However, we also found that 
teaching assistants (TA's) and team teaching are utilized in some courses (19% 
and 11%, respectively), particularly at Stanford, and most often in the 
biological and physical sciences. The presence of a significant nomber of TAs 
at Stanford is characteristic of larger, research-oriented universities. Here 
TA's read student papers and projects, grade student exams according to criteria 
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established by faculty, and provide primary feedback to students on their 
performance in the course. Our study indicates that there is a lower occurrence 
of team-teaching in our study than would be anticipated by Levine (1978). In 
contrast to his finding that 47 percent of undergraduates have taken at least 
one team-taught course , only 11 percent of the courses in our study are 
team-taught* We do not know if this finding holds true within each institution 
since our interviews were with a select group of faculty at each campus. 

Our findings do support Bergquist's (1981) contention that professional 
staff members are often "overlooked" instructional resources- Professional 
staff personnel (including administrators) are rarely called upon to develop and 
teach courses (3%) despite their rich experiences, expertise, and insight on 
particular subjects. Student peer teaching (7%) and student self -instruction 
(6%) are uncommon practices in these college and university courses as well. 
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Table 30 

Course Scheduling 

Time Offered Stanford Ithaca Mills Total 



Fall 

Winter 

Spring 

Summer 

Every year 

Two-year rotation 

Other 

Weekdays 

Weekends 

Evenings 

Other 

Contact Hours/Week 



bZ 




45 


4 


31 


88 


5 


33 


85 


103 


3 


0 


1 


1 


86 


104 


2 


0 


4 


14 


0 


1 



Stanford Ithaca 



31 


182 


0 


49 


17 


136 


0 


38 


36 


224 


4 


7 


1 


3 


40 


230 


0 


2 


0 


18 


0 


1 



Mills Total 



14.0 


1 


0 


0 


1 


12.0 


0 


3 


0 


3 


10.0 


0 


1 


0 


1 


7.5 


1 


0 


0 


1 


7.0 


3 


1 


0 


4 


6.0 


7 


2 


0 


9 


5.5 


3 


0 


0 


3 


5.0 


20 


4 


3 


27 


4.5 


1 


1 


0 


2 


4.0 


19 


11 


0 


30 


3.5 


0 


2 


0 


2 


3.0 


31 


66 


25 


122 


2.5 


1 


6 


4 


11 


2.4 


0 


1 


0 


1 


2.0 


1 


5 


3 


9 


1.5 


0 


1 


0 


1 


1.0 


1 


2 


3 


6 


Number of Weeks 


Stanford 


Ithaca 


Mills 


Total 



Regular 


0 


21 


0 


21 


15 


0 


6 


2 


8 


14 


0 


6 


35 


41 


13-17 


0 


1 


0 


1 


12 


2 


1 


0 


3 


10 


55 


0 


1 


56 


8 


0 


1 


0 


1 


. Semesters 


0 


2 


0 


2 


No Answer 


36 


70 


3 


109 



7t; 
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Table 31 

Rank and Status of Faculty Interviewed 



Lie s c r Jip l, j. o ii 


C54- anf Ar/^ 


it.naca 


mils 


lOLdl 


Pull professor 


58 


21 


ii 


90 


Associate professor 


18 


44 


9 


71 


flBD id Lai I L ^IULuodUX 










Instruc tor 


u 


7 


2 


9 


Other 


2 


2 


2 


6 


Full-time 


87 


102 


34 


723 


Part-time 


3 


4 


6 


13 


Faculty response 
Other response 


92 

2 


102 
6 


39 
1 


233 
9 



Table 32 

Who Teaches the Course 



Who Stanford Ithaca Mills Total 



Individual faculty member 68 103 39 210 

Team teaching 22 4 0 26 

Alternate days 110 2 

Teaching assistants 38 7 2 47 

Professional staff 0 4 3 7 

Peer assistants 2 9 6 17 

Student Self -Instruction 2 10 3 15 



9 
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Course Frequency 

Many faculty (51%) stated that they teach the course once a year (see Table 
33). However, nearly one-third (30%) of the faculty said that they teach the 
course each term. Stanford and Mills faculty members accounted for the majority 
of those who teach their course once a year. Ithaca faculty members accounted 
for the large majority of those who teach their course each semester. This 
finding reflects institutional differences and concomitant distinctive faculty 
priorities. Nevertheless, these differences had no apparent bearing on other 
responses in the survey. Finally, a large majority (78%) of the faculty have 
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taught their respective courses recently, either in 1991, 1990 or in 1989 (see 
Table 34). 

Where Course is Taught 

An overwhelming majority (97%) of the courses taught by interviewed faculty 
are taught on campus (see Table 35). Similar to the observations made regarding 
course scheduling, the responses to this question are characteristic of a 
traditional educational approach. 



Table 33 










Course Frequency 










Description 


Stanford 


Ithaca 


Mills 


Total 


Each term/ semester 


4 


64 


4 


72 


Once a year 


87 


30 


27 


124 


Other 










Irregularly 


2 


7 


0 


9 


Rarely 


1 


0 


0 


A 


Alternate years 


B 


2 


3 


13 


Twice a year 


2 


0 


0 


2 


Department needs 


0 


1 


0 


1 


When principles needed 


0 


1 


0 


1 


Not offered now 


3 


3 


2 


3 


Once in 5 years 


2 


0 


0 


2 


Once only 


2 


0 


1 


3 



— » < ^ 
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Table 34 

Course Last Taught by Interviewee 





Stanford 

mm Vii m m* jm 


Ithaca 


Mills 


Total 


Fall 


J / 


43 


27 


101 


Spring 






13 


99 


Summer 


u 


0 

w 


0 


0 


Winter 






0 


23 


1983 


1 


0 


0 


1 


1 HOC 

198b 


<£ 


o 


2 


4 


1986 


<l 
o 


± 


1 


8 


19b / 




a 


1 


14 






7 
* 


6 


28 


19B9 




23 


8 


65 


IQQft 


35 


67 


16 


118 




0 


0 


7 


7 




2 


1 


1 


4 


F»B7 


3 


3 


0 


6 


FaB8 


5 


5 


3 


13 


r ao? 


9 


19 


7 


35 




20 


15 


14 


49 




3 


0 


0 


3 


o£'Q r 


2 


s 


0 


7 


Sp88 


2 


2 


2 


6 


Sp89 


8 


4 


1 


13 


Sp90 


10 


67 


2 


79 


Sp91 


0 


0 




7 


niDu 


i 


o 


0 


1 


W187 


0 


0 


0 




W188 




o 


0 


r 
*) 


Wi89 


16 


0 


0 


16 


Wi90 


8 


0 


0 


8 


Other 


7 


0 


0 


7 












ii 










Table 35 










Where Course 


is Taught 








Vihere 


Stanford 


Ithaca 


Mills 


Total 


On campus 
Off campus 


88 

3 


106 
0 


41 

0 


235 
3 
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Instructional Methods 

Our findings support the notion that books, journals, newspapers, and the 
like constitute "the primary instructional materials" (Bergquist, 1981, p* 96) 
used by faculty members (90%) at most colleges and universities (see Table 36). 
Based on our research , various forms of video technology is also being used by 
faculty members (53%). For example, in the syllabi analysis, we found that 
audiovisual cassettes are common instructional methods . Although an 
instructor's lack of knowledge or skill at one time may have prevented him or 
her from using this equipment, more faculty are currently using this media 
effectively in their classrooms. In the same way, computer technology seems to 
be growing in popularity, even though a relatively small number of respondents 
(20%) included it. However, the use of the word "extensively" in the survey 
question may have prevented more participants from citing the use of computer 
technology. Simply stated, a faculty member who uses one film in her course may 
not be inclined to say she uses this computer technology "extensively" for 
classroom-related purposes. 

In addition, significant numbers of faculty interviewees explained that 
they use laboratories/studios (21%), simulated environments (17%), and 
experiential learning (20%) as ins cruet ional devices. From examining a sample 
of course syllabi, for example, we learned that one instructor required students 
to spend several days at a physical therapy zlinic; other faculty members 
required the completion of weekly laboratory homework assignments. Many of the 
instructors who include these instructional methods are in the physical, 
biological, or behavioral sciences, engineering, and the performing arts. 

At this point, it may be instructive to reiterate an earlier point. Most 
of the faculty who were interviewed considered the purpose of teaching to be 
"teaching 'thinking processes' ." A certain confusion existed regarding faculty 
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members' elated teaching purposes and their evaluation of students' cognitive 
development . A similar observation can be made now. According to Bergquist 
^1981), content-based teaching, or, what Axelrod (1973) refers to as the 
didactic mode, includes the following instructional methods: lecturing 
(individual faculty member teaches class— Question #9), reading (print 
medium— Question #13), question and answer exercises (tests—see Question #15), 
and audiovisual instruction (video technology—Question #13). 

In contrast, interaction-based teaching, or what Axelrod terms the 
evocative mode, includes these instructional methods: team teaching (Question 
#9), laboratory/studio (Question #13), simulations (Question #13), 
seminar/discussion, case study, role playing, and in-class discussions (class 
participation—see Question #15). The last few teaching methods cited here 
rarely, if at all, appeared in the comments of those interviewed or in the 
analysis of sample syllabi from the courses. 

Once again, a certain incongruence exists in many faculty members 1 
perceptions. Stated another way, many of those interviewed see themselves as 
"interact ion-based," or "evocative" teachers whose expressed primary purpose in 
the learning process is to teach students "tninking processes." In reality, 
however, these faculty members fulfill their roles as "content-based," or 
didactic" teachers whose day-to-day actions are to "transmit knowledge" to 
students. The point is not to evaluate the "goodness" or "badness" of the 
evocative or didactic modes of teaching. Rather, the important issue is to 
affirm that teaching involves much more than simply "talking about what one 
knows" or "managing the classroom effectively." As Shulman (1987) explains, 
exemplary teachers continually weave together threads of what they know, how 
they know, and how they articulate what and how they know to students. From 
this perspective, dynamic teaching "begins with an act of reason, continues with 
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a process of reasoning, culminates in performances of imparting, eliciting, 
involving , or enticing, and is then thought about some more until the process 
can begin again" (Shulman, 1987, p. 13). 



Table 36 

Instructional Methods Used 



Resource 


Stanford 


Ithaca 


Mills 


Total 


Print medium 


82 


97 


38 


217 


Audio technology 


11 


28 


7 


46 


Video technology 


42 


65 


22 


129 


Computer technology 


23 


22 


4 


49 


Labs/studios 


17 


24 


10 


51 


Simulated environment 


10 


26 


5 


41 


Experiential learning 


15 


28 


5 


48 


Field trips 


6 


4 


6 


16 



Course Planning 

A large majority of faculty members (90%) explained that they are solely 
responsible for setting the goals and content of their courses (see Table 37). 
This finding is consistent with the course planning practices of mos^ colleges 
and universities. According to Bergquist, "faculty-planned curricu3"ms are 
pervasive and respond to the legitimate need of students for clear and detailed 
information about the courses they will take and the courses of study they will 
follow for particular careers" (1981, p. 144). Faculty members' responses to 
this question also coincide with the instructional purposes and methodologies 
that they utilize in the classroom. 
How Learning is Assessed 

Most faculty members use multiple measures of evaluation to assess student 
learning (see Table 38). The sample syllabi analysis confirms survey data that 
some combination of tests, papers, projects, and class participation/ attendance 

S3 
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typically serves as the criteria for student evaluation. Tests are the roost 
common assessment tool, being cited by a large majority of the faculty who were 
interviewed (87%). From the sample test analysis, we found that course tests 
most often are objective in nature, in that they consist of multiple choice, 
true/false, matching, and short answer questions. Essay questions and problems 
or computations also were present in the tests sainple, but not as frequently and 
with less comparative weight. Other coasaon evaluation tools used by faculty 
members in the study are papers (43%) and projects (26%), which may be expanded 
to include exercises such as homework, lab assignments, and workbooks. 
Who Assesses Learning 

An overwhelming majority of the respondents (97%) stated that faculty 
members are the evaluators of student learning (see Table 39). Clearly, this is 
a common practice at most colleges and universities. In addition , we found that 
TAs also do a considerable amount of student assessment (19%), including reading 
students' examinations and assigning grades. This corresponds with an earlier 
finding that TAs are involved in the teaching process, particularly at research 
institutions such as Stanford. 

A potential problem exists, however, when students are not involved in the 
assessment process. Bergquist notes: M the authoritative role of the 
teacher is potentially exaggerated. Pleasing the powerful 'other 1 (the 
instructor, or the TA) may replace learning as the student's motivation" (1981, 
p. 242). According to Bergquist, this issue is resolved when students are 
partial or full participants in the evaluation process. The picture suggested 
by our data, however, reveals only slight student involvement in the assessment 
of learning. 
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Table 37 
Course Planning 



Who Plans Stanford Ithaca Mills Total 



Faculty 88 
Faculty & students 2 
Students 1 



93 36 217 

13 4 19 

0 12 



Table 38 

How is Learning Assessed 



Medium Stanford Ithaca Mills Total 



Tests 


84 


98 


30 


212 


Papers 


33 


48 


23 


104 


Projects 


15 


38 


9 


62 


Presen tat ion/ Performance 


8 


37 


19 


64 


Other 










Homework 


15 


3 


3 


21 


Lab assignments 


5 


6 


0 


11 


Participation/attendance 


1 


11 


4 


16 


Workshops 


0 


1 


0 




Workbooks 


0 


2 


1 




National teaching exam 


0 


1 


0 




Rehearsal 


0 


1 


0 




Skill competency checks 


0 


1 


2 




Group presentation 


0 


1 


0 




Honors work 


1 


0 


0 





Table 39 

Who Assesses Learning 



Who Assesses Stanford Ithaca hills Total 



Faculty 


87 


106 


41 


234 


Student peers 


6 


7 


5 


18 


Student self-appraisal 


3 


11 


10 


24 


External criteria 


0 


3 


0 


3 


Other 










TAs/GAs 


42 




0 


45 


Homework 


4 


0 


0 


4 


National teaching exam 


0 


2 


0 


2 


External sponsor 


2 


0 


0 


2 


Other faculty 


0 


1 


0 


1 
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CONCLUSION 

Coursework from the transcripts of two samples of graduating seniors at 
Stanford University, three samples at Ithaca College, and two samples at Mills 
College were associated with gains in nine types of learning measured by the 
Graduate Record Examination. Most of the 9 item- types proved to be reliable, 
discrete and valid measures of general learned abilities. College officials can 
now examine these types of learning to determine if they should receive greater 
or less attention in the respective curricula of the three institutions. The 
Analogies and Antonyms questions test abilities frequently valued in the social 
sciences. Graduates are often asked to demonstrate these abilities on the 
Millers Analogies Tests, as well as other standardized graduate and professional 
examinations ured for admissions and fellowship awards. Similarly, the ability 
to interpret data is fundamental to the physical and social sciences, as well as 
to applied fields, such as Engineering. 

In most samples Reading Comprehension, Quantitative Comparisons and 
Analytic Reasoning proved to be types of general learned abilities where large 
changes occurred. The CAM research indicated clearly that this is where the 
majority of general student learning occurs in the coursework in which students 
coirononly enrolled. Faculty who teach these courses encourage the development of 
Analytic Reasoning and Reading Comprehension through the syllabus, assignments 
and examinations they require of students. Less clear is the development of 
Quantitative Comparisons abilities. Faculty were unable to idenU y what the 
types of skills and abilities were required of students to succe ;sfully answer 
these types of questions. 

The faculty interviews tended to corroborate the CAM findings, with the 
previously noted exception of Quantitative Comparisons. CAM identified courses 
that were linked to gains in one or more abilities represented on the GRE 
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examination. Faculty were presented with examples of all 9 types of questions, 
so they had greater opportunity to pick item-type examples not linked to their 
course by the CAM procedure- Nevertheless, in most cases faculty did pick the 
GRE item- type with which their course had been identified and were only too 
happy to articulate how they strived to develop abilities. Few, if any of these 
faculty planned their course to develop any of the abilities examined. Still, 
there were underlying themes, skills or habits which faculty sought to develop 
in the vay that students viewed and analyzed knowledge. Both the CAM findings 
and the faculty interviews do not leave us with a clear association between 
specific disciplines and general learned abilities. The conventional wisdom 
that only math classes develop math skills and only English classes develop 
reading comprehension abilities was not affirmed. The groups of courses, no 
matter whether they are derived from the Cluster Analytic Model or the faculty 
interviews, suggested that time-honored curricular notions of the role of broad 
disciplines to the structure of knowledge is fractured and incomplete. General 
learned abilities cut across fields. Many majors, programs and subject areas 
have their own philosophy course, their own history course, their own 
quantitative and qualitative methodology courses and the like. The curriculum 
is more like a fabric of interwoven threads than a system of sequences and 
linear relationships between subjects. Admittedly, these are emerging 
impressions, and far more examination of the relationship between what students 
specifically studied and what they precisely learned is very much needed. Only 
through such efforts will curricular reform go beyond the spasmodic review and 
cathartic emoting of academic leaders and faculty committees. 

Curriculum at a liberal arts college, a comprehensive college, or a 
research university is necessarily complex, consisting of hundreds or thousands 
of course choices for students. These course choicer represent the variation in 




the incoming ability of the students, the expanding knowledge base in the 
disciplines and fields of study and the dynamics of faculty experimentation and 
reform of the curriculum, m these forces make identifying and selecting 
appropriate coursework problematic. Faculty did not have a current working 
knowledge beyond their immediate field of study upon which to base specific, 
relevant advice to students regarding coursework to improve cognitive 
abilities. Given hundreds or thousands of courses in the undergraduate 
curriculum and given that each is designed to produce a distinct contribution to 
student learning, it is perhaps impossible for any single individual to provide 
such timely appropriate advisement to students. Yet, beyond the complexities 
of the curriculum, faculty have yet to develop a professionally responsible 
basis for offering such advice. When students must choose between several math 
or science courses to fulfill their general education requirements, a faculty 
member's recollection of their own undergraduate education, then stereotypic 
notions of broad subject areas, thumbing through the college catalog, or making 
recommendations based on the rumors and off-hand comments passed by other 
faculty or other students is not a sound basis for constructing an educational 
program. Given the promise of the Cluster Analytic Model to isolate coursework 
associated with gains in learning for a particular group or student, a more 
exact, well-informed system of student guidance can be constructed. 

While there is a critical need to improve the quality of information and 
the faculty skill in advising students, there is a commensurate need to better 
align what faculty intend to teach with their instructional and classroom 
examination practices. Critical to this improvement at research universities is 
the recognition of teaching assistants as legitimate custodians of much of the 
lower division undergraduate curriculum. TAs read students' papers and 
projects, assist students with their immediate learning problems, and grade and 




provide feedback to students aboat their performance on examinations 

At all three institutions examined , students at or above the mean were more 
likely ^ enroll in coursework associated with improvement in student learning 
(Ratcliff, 1990a, 1990b, 1990c, 1990d) . At all three institutions, faculty 
overestimated student cognitive skills in such areas as Sentence Completion and 
Regular Mathematics. At both institutions faculty had clear visions of 
themselves as evocative teaching promoting analytic reasoning and developing the 
thinking process. While the CAM did show that the greatest proportion of 
student improvement in general learning did occur in Analytic Reasoning, course 
syllabi, course requirements, and classroom examinations continued to stress 
knowledge transmission, didactic instruction, and lower levels of cognitive 
learning. 

Given that we interviewed faculty who taught coursework clearly associated 
with student improvement in general learned abilities, we conclude that most of 
the best teachers and classes of these institutions are still trying to develop 
students 1 basic knowledge of terms, concepts and theories rather than engaging 
students in critical analyses and problem- solving activities. Only by 
increasing faculty expectations for higher order analytic abilities will bring 
about such changes in the curriculum. 

To improve general education, we must close the gap between intentions and 
practice. We can do so through systematic assessment and analysis of coursework 
patterns. This information, in turn, can be used to launch new, better- informed 
bases for student advisement, teaching, and learning. 
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APPENDIX A 
Letter Requesting Faculty Participation 




LETTER iQUESTING FACULTY PARTICIPATION 



March 28, 1990 



Dear Colleague: 

I am writing to enlist your support and cooperation in an ongoing research 
project. For the past three years , the University has been cooperating in 
a major research effort being carried out by Dr. James Ratcliff , formerly 
of Iowa State University and currently the Director of the Center for the 
Study of Higher Education at The Pennsylvania State University. Dr. 
Ratcliff and his associates have been investigating the relationship 
between course-taking patterns and gains in student scores on nine 
item- types measured by the GRE. 

Funded initially by the U.S. Department and subfife^niently by the Exxon 
Foundation this exploratory research uses transcript analysis to identify 
these relationships. During 1987 and 1988, two Stanford student groups 
participated by taking the GREs at the project's expenses. It is their 
transcripts that have been analyzed . 

The initial analysis has identified 108 Stanford courses that are of 
interest to the project's researchers. These courses are associated with 
particularly high increases in student scores on one or more of the nine 
item- types measured. In order to understand better the nature of these 
courses, the researchers have requested the opportunity to interview the 
faculty members teaching these courses. 

The purpose of this letter is to alert you that during the next few days, 
you can expect a call on behalf of the project researchers requesting the 
opportunity to meet with you. The interviews have been pilot tasted and 
will take approximately twenty minutes, to be scheduled at your convenience 
sometime between April 4 and 10. Your participation is completely 
voluntary . 

The interview is structured in two parts. First, you will be asked two or 
three questions regarding the nature and design of your course and your 
teaching methods* Second, you will be asked to discuss a series of 
advising questions related to sample GRE questions representing each of the 
nine item- types, e.g., what couises would you reconmiend to a student 
needing to develop that skill or krowledge, and is your course designed to 
develop that learned ability? 

It is important here to emphasize that this is not an evaluation of the 
Stanford curriculum , but rather an exploratory investigation into 
associations that have been preliminarily identified between course taking 
patterns and student improventent on the GRE item types. The objective is 
better understanding of the effects of general education course work and 
curricular choices on student learning as measured by tKi GRE. This group 
of researches has completed similar work at Evergreen State, in 
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Washington, and Georgia State and is ^^.^.^'J^^oS It 
Ithaca College in New York, and Mills College, in addition to Stanford. It 
is hoped that, this research will provide some iimights that may help 
students better plan their programs in order to enhance strengths and 
overcome deficiencies in these areat of learned abilities. 

If you have any questions, please don't hesitate to raise them with the 
researchers when Uu Y call, or you can contact Virginia E. Ramos directly 
at 3^550 5 Sou oo not receive a call, it will be because their quota of 
forty interviews has been filled by prior calls. Again, remember that your 
participation, if requested, is complete voluntary. Thanks much for your 
cooperation . 

Sincerely, 



Sally Mahoney 

Registrar & Senior Associate Provost 
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DIFFERENTIAL COURSEWORK PATTERNS PROJECT 



FACULTY SURVEY 



James L Ratcliff 
Professor and Director 

and 

Elizabeth A. Jones 
Research Assistant 



Oantor for th# Study of Hlghtr Education 
THo Pennaylvanls State Unlvorolty 
408 South Alltn Street, Suite 104 
University Park, Pk 1S6O1-02O2 
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INFORMATION FOR COURSE 



What percentage of your final evaluation of etudaata is based on aach of 
the following araas? 

Pareant Evaluation of students in the area of i 

a. Cognitive development 

_ ______ b. Affective dovolopaoat 

c. Psychomotor development 

100% TOTAL EVALUATION 



Rata tha importance of the following eoonltlva development aroaf in yovr 
overall .valuation of students. 

Very important .... 5 

Xisportant 4 

Neutral 3 

Unimportant 3 

Vary Unimportant . . 1 



Evaluation of students' ability tot 
1 2 3 • 5 a. Oain basic knowledge, language or terms. 



1 2 3 4 5 



b. Understand concepts, theories, and trends of the 
field of study. 

1 2 3 4 5 c. Perform, act out, demonstrate skills, involved 

in tha field of study. 

d. Distinguish between facta and inferences, 
recognise tha coaponents of an argument, theory, 
etc. 

e. integrate learning froa different areas into an 
original idea or problem solution. 

f . judge tha worth, value, or merits of something 
based on specified criteria. 



1 2 3 4 5 

1 2 3 4 5 
1 2 3 4 5 
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Please rank order your primary purpose in teaching tbis course (1 is 
cloeeet to your primary purpose, 4 is further from your primary 
purpose) s 

Za my couras, X try to ... 

Btak 

a. Hava etudente learn courso content. 



b. Hava student a loan a particular perspective regarding 
course content . 

c. Taach students tor to laa ra- 



ft. Assist atudanta to incorporate certain akilla and/or 
knowledge into thair daily, personal lives. 



Primarily, this courao ia of farad ... (Mark all that apply) 

a. Pail torn 

b. Winter tans 

c. Spring tans 

d. Summer term 

a. Every year 

f • On a two year rotation 

g. Other (apecify) _ 

h. On weekdays 

i . On weekends 
j * Evening a 

k. Other 

1. Number of contact hours par weeks 

m< Number of weeks (if not equal to the regular course calendar)! 



Pleaaa provide the following information about youreelf . Tour rank ia 



a. Pull professor 

b. Associate professor 

c. Assistant professor 

d. Instructor 

m. Other (specify) 

You are contracted aa . . . 

f. Pull-time 

g. Part-time 

Your primary responsibility is • . . 

h. As faculty 

i. Other (specify) ^ == ^ 
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ORE Bij«PiB QUESTIONS " — 

SJSJStSS?™ 11810 " "" ** adl,l » 10 «*««taad « writtan pas.ag. fro. .everal 

JtlIf!f 0,l ";^ BaCl1 »;!»•«• *■ W follow by questions based on its 

EES ali^JSf^t? P !' M9 *' chooM **• te each Ration. 

±£u«/iJ s:rjsi.s;! ^ ■ pMMg * 0,1 **• hmau ° £ ~ 

Initially the vinaver theory that Malory »a sight muetf, one* tbouoht to b. 
fimdamentally unified, nr. la fact eight looaV-ad«^ri.^d«ca^bott . 

°' *»■ •» nnpleaeant shock. JSaVirVthaory coafortably 

explained away the apparent contradiction* of chronology and made each romance 
lodapondontly satiafying. It waa, howmr, diaagroa^ to find that wJat^ad 
boon thought of as ona book una no* eight hooka. Part of tU^nouTn, 
the natural reaction to tho disturbance of act id.... Ifararth.lalf? I^ iow 
after lengthy consideration of tho theory a raflnad hot legitiaaJJ ' 

otr^r^Jt ?:"n^ n ^ t : V ! id CO ri?* ioa «- •** romance, are only 

ona work. It la not quite a matter of dlaagraaiog with tha thaorr of 

^dependence, but of rejecting ita implications! that tho roaancee nay be 

taken is any or no particular order, that they have no cumulative of fact and 

that tbay are aa separata •« tha aorki of a 2d.ru uovaliTt. ' *"* 

3.1 She primary purpoaa of the paaaage la tot 

£*)* Discutt the validity of a hypothesis . 

(B) Summarise a system of ganaral principles. 

(C) Propoae guidallnea for future argument. 

fS! £?f pul * t " o 0 * 1 **^™ for acceptsnee of an interpretation. 
' ' """I sccusations about an apparent contradiction. 

3.2 It can be inferred from the paaaage that the author believes which of 
the following about Malory 'a works? ox 



XI 



r' Sriunn-SS^f'S liak " hmtummD " d tha roaances. 

ona wort. Imnc " *" obscurad when tbay are takan aa 



1X1 * ^L!^ Dt f*?f Cti ?" cl »«>nology among the romances a . i eSB 
Important than thalr ovar-all unity. 

(A) I only (B) liz only (C)* I and III only 
(D) IX and IXI only (E) I, 11, and III 



Questions rafarring to ORE aampla questions on Raiding Comprcbeeeion 

Sa'ona* SSJJ? *** * 8<,Cb 9tuAnU *» "•-•ring question, similar to 



*es _ Mo 



If yes, how? 



17b, 



17c, 



JLli!^ U ?!!° t .^* ia . y0ur cl «"> cama to you to ask for advice on how to 
1a ?i OF * th# ,* bi JJ ty to thLt ««w«tion bje* would you decida what 

advica to give him or her? 

IfiK would you dacide what classes to recommend for the etudent? 

If a student wished to improve bia/ber abilitias relative to the above 

SSSrsS TxXeJ) 0 ' you recommend? (OBTAIN SPECIFIC 
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sentence COMFXETZGS — Zdentlf ying words or phrases which both logically and 
stylistically cenplete the swan leg of a sentence* 

Directions i Bach sentence below has mo or two blunks, each blank indicating 
that eoaething has boon onittod. Banoath tho aontonco am fire lottarod words 
or sata of words. Chooso tho word or aot of words for oaoh blank that bast 
fits tho weaning of tho ao nto n c o as a whole. 



1.1 Tho ahoor bulk of data fro* tho nass nodi* soma to overpower ns and 
drive us to ______ eccounta for an oasily and readily digestible 

portion of news. 

(ft) Insular (B) Investigative <C)* Synoptic 

(D) Subjective (E) Sensational 

1.2 Tho salo of Alaska was not oo nuch aa American coup as n natter of 

for an inperial Russia that waa short of cash tnd unable 

to ita own continental coastline. 

(A) Negligence* .fortify (B) Cuf to*, .naintala 

(C) Convenience. .stabilise (D)* Expediency. .defend 
(E) Exigency , . reinforce 



Questions referring to ORE sample questions on Sentence Completion 

19a. Doe a the course yon teach aid students in aaeweriag questions sinllar to 
the ones above? 

Yes _____ Ko If yes, howf 

19b. Zf a student (not in your class) cane to you to ask for advice on how to 
develop the ability to answer this question, tot would you decide what 
advice to give him or her? 

gSM would you decide what classes to ro c onaond for the student? 

19c. Zf a student wished to inprove his/her abilities relative to the above 
sanple question, what course (s) would you reconnend? (OBXfcXV SPECIFIC 
COURSE TITLES) 
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ORB SAMPLE QUB8TIGHS 



DATA IKTERPRETATIOJI — Selection of data for answering questions 
Directions i Sao following question refers to the following graph. 

RESULTS OP A SAMPLE OF VOTERS 
IN DISTRICT X 




Candidstt 



The graph above shows bow a sample of 10 different voters (vortical axis) 
voted for 5 different candidates (horlsontal axis). Bach voter voted for 
either one or two of the five candidates. (Ho voter voted twice for the sane 
candidate.) She two candidates receiving the nest votes were the winners. 
The sanpla constituted 9 percent of these in the district who voted, and ths 
number of votes in the district for osch candidate was in the sane proportion 
as the number of votes in the saaple for each candidate. 

7.1 Nhat fraction of the total number of votes cast did the two winners 
receiver 



<*)• 11 (B) 11 (C) JL (D) _i_ (B) _1 
18 30 2 3 10 

Questions referring to ORE sample questions on Data Interpretation 

21a. Does the course you teach aid students in answering questions similar to 
the ones above? 

»o if yes, how? 

21b. Xf a student (not is your class) came to you to ask for advice on how to 
dsvelop the ability to answer this question, hpw. would you doc ids what 
advice to givo him or her? 

fissf would you decide what classes to recommend for the student? 

31c. Xf a student wished to improve his/her abilities relative to the above 
sample question, what course (s) would you recommend? (OBTAIN SPECIFIC 
COURSE TITLES) 
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QUE SAMPLE OUEfJTJONS 

LOGICAL RZASCHIWO ~ Recognising relationships isoBf arguments or parts of 
argument a* 

Directions* sack question or group of questions la based on a passage or aat 
of conditions. Za answering acme of the qu est io n s, it nay bo useful to draw a 
rough diagram, tor each question, eeleet tha boat answer choice given. 

Dormitories ranga froa two to sin itorloa la height. Xf a dormitory root ia 
abova the second floor # it baa a firs escape, 

8.1 Zf tha statements above aro true, which of tba following amat also bo 
truo? 

(A) Second- floor dormitory rooaa do not have firs escapee. 
(») lhlrd-floor dormitory room* do not have firm escapes. 

(C) Only dormitory rooms abovo tba aooood floor have firm aecapss* 
(P) # rourth- floor dormitory rooaa have firs escapes. 

(B) Soma two-story dormitorioa do mot have firo escapes. 

Unlike other forma of narrative art, a play, to bo successful, mast giva 
pleasure to Itf immodiata audiaaco by rof looting tba coacarma and values of 
tbat audience. A aoval can achieve success ovar moat**s or even years, bat a 
play mast ba a bit or pariah. anoooaafmX drama of tba Roatoratioa period, 
tharaforo ia a good index to tba typical taataa and attltmdaa of ita tine. 

8.3 Tha author of tba paaaaga abovo us sum* s tbat 

(A) Playa writtaa for iss to ration eudleacea do not appaal to modern 
audiences. 

(B) Plays ara aoporior to aovola aa a form of narrative art. 

<C>* Restoration amdianeaa vara representative of tba mhola population 
of tbair time. 

<»> Flaygoars and novel roadart ara typically distinct and axolusiva 
groups • 

(E) Restoration drama achieved popular success at tba expense of 
critical succees. 



Questions referring to ORE sample questions on Logical Reasoning. 



23a* Does tba course you teach aid students in answering questions similar to 
tha ones abova? 

Yes Ho If yes, ho*? 



23b. Zf a student (not in your class) com to you to ask for advice on hem to 
develop tba ability to answer this question, fegg would you decide what 
advice to give him or bar? 

8fiV would you dacida what clasaas to recomaend for tha student? 



23c Zf a student wlabad to improve his/her abilitiea relative to the abova 

sample question, what course (s) would you re c emend? (OBTAIN SPECIFIC 
COURSE TITLES) 
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ORB SAMPLE QOESTXOMS 
- Vocabulary; knowledge of opposltes 



Direction! i Back question below consists of • word printed la capital 
lettere, followed by five lattarad words or pbraaoa. Choose the lettered word 
or phrase tbat la aost aearly opposite is wining to tba word is capital 
latter a. 

Sinca some of tba questions require you to distinguish fins abadas of weaning, 
ba sura to considar all tba cboicas bafora deciding which one ia beat* 

4.1 SERRATED I 

(A) without joints (B) Without folds (C)* Without notch. ■ 
(D) Variegated (B) Mutilated 

4.2 FLEDGLING I 

(A)* Experienced practitioner (B) Successful competitor 
<C) Reluctant volunteer (D) Recent convert (E) Attentive llatener 



Questions referring to ORE staple questions 9a Antonyms 

25a. Doss the courae you teach aid etudeata in answering questions siailsr to 
tba ones above? 

*•» *> If yes, bow? 



25b. Xf a student (not in your class) came to yon to ask for advice on bow to 
dsvelop the ability to answer this question, how would you decide what 
advice to give bia or bar? 

How would you decide what classes to r o coaaou d for the student? 



25c. If a student wished to iaprove his/her abilities relative to the above 
staple question, what course(s) would you recoaaend? (OBTAIN SPECIFIC 
COURSE TITLES) 
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APPENDIX C 

Procedures for Syllabi & Tests Examination 



In 4 
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PROCEDURES FOR SYLLABI & TESTS EXAMINATION 



We requested a syllabus and a test from each interviewee as part of our 
data collection in an effort to underestand, as fully as possible, what occurs 
in the course. The examination of the syllabi and tests followed several 
steps. First, we determined what we actually were able to collect on site (see 
Table 1). 

Second, we selected a random sample of 25 from the "Both Syllabus and Test 
With Purpose" category--20 from Ithaca and five from Stanford— that represented 
a wide array of courses. We later determined that two of the samples were 
unusable, giving us a working sample of 23 sets of syllabi and tests* The 
sample was drawn f ran this category because we wanted to be able to compare the 
stated purposes of a course (including the expected levels and types of 
learning) with how students are assessed). 

Third, We devised two protocols: one to collect information from syllabi, 
ana the other to gather information from the tests to examine the levels of 
learning expected of students taking the course (using Bloom's taxonomy), and 
the types of learning expected of students taking the course (utilizing the nine 
item-types from the GRE General Test). In addition, both protocols solicited 
other kinds of useful information such as instructional methods used, how 
students of useful information such as instructional methods used, how students 
are evaluated, and the nature of the tests (i-e., multiple choice, essay, etc-)- 

Fourth, we developed a strategy to analyze the syllabi and tests using the 
respective protocols. Previous research on course planning (NCRIPTAL, 1988) was 
extremely valuable in this process. The plan was as follows: 

1. Choose 3 raters. 

2, Develop a Protocol Explanation Sheet (see Figure 3). 
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3. Each rater will review a Llabus and a test, and the syllabus and 
test protocols to become iamiliarized with the task. 

4. Each rater will evaluate seven sets of syllabi and tests using the 
appropriate protocol. 

5. Tho raters will meet to discuss issues/problems that emerged in the 
rating of the first seven sets of syllabi and tests. 

6. The raters will evaluate the remaining 16 sets of syllabi and tests. 

7. The raters will reevaluate the first seven sets of syllabi and tests. 

8. The findings from each raters' evaluations will be tabulated and 
compared . 

9. The raters will meet to discuss the similarity and dissimilarity of 
reporting . 

10. The final findings will be tabulated, analyzed, and incorporated into 
the f inal report . 

Fifth, we examined the sample of 23 syllabi and tests following the above 
plan. Two particular issues deserve mention. The first issue relates to the 
importance of "regrouping" after the raters had evalua' d seven sets of syllabi 
and tests. This provided vital feedback regarding working definitions, raters' 
perceptions, and how to deal with various expressions contained in the syllabi 
and tests. As a result cf this debriefing, the raters generated a list of 
"tips" to be used while completing the remaining sets of syllabi and tests (see 
Figure 4) . 

The second issue involves synthesizing the coimpleted protocols. Each 
rater's evaluations of each of the 23 sets of syllabi and tests were tabulated. 
Then, focusing on the use of Bloom's taxonomy and the nine item-types of the GRE 
General Test, we determined the points at which disagreement between raters 
existed (defined as those instances in which at least one rater said that the 
level or type of learning was neither illicit nor explicit, and at least one 
rater said that the level or type of learning was either implicit or explicit). 
Accordingly, we found that, for each protocol set (syll^us and test of one 
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course), disagreement existed in an average of approximately 13 (X « 12.7) of a 
possible 30 rating. Within these 13 areas of disagreement , wide discrepancy (at 
least one rater said that the level or type of learning was neither implicit or 
explicit and at least one rater said that the level or type of learning was 
explicit) occurred an average of approximately 5 times (X * 4.7). 

The raters met to discuss each of these inconsistencies and examine syllabi 
and tests together. We found that, in almost every case, evaluative 
disagreement between raters was a result of one of two issues: (1) overlooking 
a statement in a syllabus or a question in a test, or (2) being unusually 
liberal in evaluating the level or type of learning- It should be noted that 
perception or bias was seldom the primary factor in explaining discrepancies . 
In fact, after the raters' discussions, only 4 of the original 292 total 
disagreements remained, all of which were due to differences in rater 
perception. 

Finally, the 23 sets of syllabi and test were analyzed to produce frequency 
distributions and simple comparisons. Several conclusions based upon this 
analysis were incorporated into the differential coursework patterns faculty 
survey final report. 

PLAN TO ANALYZE SYLLABI AND TESTS 

1. Esto ish 2 or 3 raters. 

2. Each rater will read through the syllabus and test protocols, the protocol 
explanation sheet, and a sample syllabus and test to become familiarized 
with the task at hand. 

3. Each rater will evaluate seven (7) sets of syllabi and tests using the 
appropriate protocols. 

4. The raters will meet to discuss problems and determine the similarity or 
dissimilarity or reporting. 

5. The raters will evaluate the remaining eighteen (18) sets of syllabi and 
tests. 
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6. The raters will reevaluate the first seven (7) sets of syllabi and tests. 

7. The results will be tabulated, analyzed, and incorporated into the final 
report . 

8. The evaluation procedures used here will be included as part of the report. 



Appendix C— Table 1 

Information Collected at Saraple Institutions 



Stanford Ithaca Total 



Syllabus Only 
With purpose 
Without purpose 

Test Only 

Both Syllabus & Test 
With purpose 
Without purpose 

Neither Syllabus Nor Test 



2 6 8 

5 2 7 

11 4 15 

12 50 62 
29 20 49 

26 12 38 
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Figure 1 

Protocol for Syllabi Analysis 



Institution 

Instructor 

Course 

1- Does the syllabus explain the purposes/objectives of the course? 
Yes No 

2. According to the syllabus , indicate the degree to which the following 
levels of learning are expected from students as follows: 

"0" — the level of learning is neither stated or implied in 
the syllabus 

"1" the level of learning is implied but not stated explicitly 
in the syllabus 

"2" — the level of learning is stated explicitly in syllabus 

Gaining basic knowledge, understanding concepts, as follows: 

Knowledge Application 

Comprehension Analysis 

Synthesis Evaluation 

Interest, appreciation, attitude of learning in class 

(e.g., class participation) 

Development of certain skills, abilities, competencies 



hcco -fijng to the syllabus, indicate the degree to which the following 
types of learning are expected in this course as follows: 

"0" this type of learning is neither stated nor implied in 
the syllabus 

"1" this type of learning is implied but not stated explicitly 
in the syllabus 

"2" this type of learning is stated explicitly in the syllabus 



Analogies Antonyms 

Sentence Completions Reading Comprehension 

Quantitative Comparisons Data Interpretation 

Analytical Reasoning Logical Reasoning 

Discrete Quantitative (Mathematics) 



Does the syllabus describe course requirements or areas of evaluation? 

Yes No (Please weight below.) 

Tests Papers 

Discussion/Debate Labs 

Attendance Presentation/Performance 

Homework/Exercises Projects/Other 

Are the texts for the course mentioned in the syllabus? 
Yes No 

Does the syllabus include a weekly schedule of activities? 

Topics Readings 

Assignments Other 

Does the syllabus describe the inclusion of various instructional methods 
Yes No 

Print media Audio technology 

Video technology Computer technology 

Labs or studios Simulated environments 

Field trips Experiential learning 

Case studies Other 



llll 
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Figure 2 

Protocol for Test Analysis 



Institution 

Instructor _ 

Course . 

1. What types of questions does the exam include (weightecj? 

Multiple choice True/False 

Essay Short Answer 

Sentence completion Computations/Problems 

Other 



2- Indicate the extent to which this test measures the following GRE 
item-types as follows: 

"0" — this item-type is neither explicitly or implicitly 
teste. in this test 

"1" — this it '3m- type is implicitly but not explicitly tested in 
this test 

"2" this item-type is explicitly tested in this test 



Analogies Antonyms 

Sentence Completions Reading Comprehension 

Quantitative Comparisons Data Interpretation 

AnalyMcal Reasoning Logical Reasoning 

Discrete Quantitative (Mathematics) 



til 
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3. Indicate the extent to which this test measures the components of Bloom's 
taxonomy as follows: 

"0" — this component of Bloom's taxonomy is not explicitly or 
implicitly tested in this test 

"1" — this component of Bloom's taxonomy is implicitly but not 
explicitly tested in this test 

"2" — this component of Bloom's taxonomy is explicitly tested in 
this test 

Knowledge Comprehension 

Application Analysis 

Synthesis Evaluation 
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Figure 3 

Explanation of Protocols for Raters 



To ensure that we are using similar starting points as we rate syllabi and 
tests from Ithaca and Stanford, a brief explanation of the components of 
Bloods taxonomy and the nine item- types included in the GRE General Test 
are necessary. Please adhere to these explanations as you complete the 
protocols . 



Components cf Bloom 1 s Taxonomy 

Knowledge . The recall of specifics and universals, the recall of methods and 
processes, or the recall of a pattern, structure, or setting. 

Comprehension . Ability to know what is being consminicated and the ability to 

use the material or idea being communicated without necessarily relating it 
to other material or seeing its fullest indications. 

Application . The use of abstractions- -general ideas; rules or procedures; 
generalized methods; technical principles, ideas, and theories— in 
particular and concrete applications. 

Analysis. The ability to breakdown conanunication into its constituent elements 
or~parts such that the relative heirarchy of ideas is made clear and/or the 
relations between the ideas expressed are made explicit. 

Synthesis . " » putMng cogether of elements and parts so as to form a whole. 

Arranging and c nbining things in such a way as to constitute a pattern or 
structure not clearly there before. 

Evaluation . Qur>- itative and qualitative judgments about the extent to which 
material ai^ methods satisfy criteria. Use of a standard of appraisal. 



Nine GRE Item-Types 

Analogies . Ability to recognize relationships among words and the concepts they 
represent and to recognize when these relationships are parallel. 

Antonyms . The ability to reason from a given concept to its opposite. 

Sentence Con^letion . The ability to recognize w rds or phrases that both 
logically and stylistically complete the meaning of a sentence. 

Reading Comprehension . The ability to read with understanding, insight, and 
discrimination; ability to examine a written passage from several 
perspectives. 

Quantitative Comparisons . The ability to reason quickly and accurately about 
the relative sizes of two quantities or to perceive that not enough 
information is provided to make such a decision. 
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Discrete Quantitative . The ability to use basic mathematical knowledge in 

answering questions in which all the information required for answering a 
question is provided. 

Data Interpretation . The ability to synthesize information, to select 

appropriate data for answering a question , or to determine that sufficient 
information for answering a question is not provided (similar to reading 
comprehension in the verbal section). 

analytical Reasoning , The ability to understand a given structure of arbitrary 
relationships among fictitious persons, places, things , or events, and to 
deduce new information from the relationship given. 

Logical Reasoning . The ability to understand, analyze, and evaluate arguments, 
which includes recognizing the point of an argument and the assumptions on 
which an argument is based; drawing conclusions and forming hypotheses; 
identifying methods of rgument; evaluating arguments and 
counter-arguments; and, analyzing evidence. 



Figure 4 

Tips to Follow with Protocols 



1. Bloom's taxonomy is a heirarchy. Therefore, if you enter a "2" for one 
category , everything above it has to be a "2" also. You cannot have larger 
numbers after smaller ones. 

2. Almost always, knowledge and comprehension go together so a n 2 %i for one 
means a "2" for the other. 

3. In test protocol, be conservative; especially for antonyms, sentence 
completions, and reading comprehension. 

4. In test protocol , if matching is in test, put a "2" for the item-type 
Analogies. 

5. Data Interpretation on test protocol requires sane kind of chart in the 
test . 

6. In test protocol, Analytical Reasoning involves How and Why questions 
mostly, and builds on what has been provided in terms of deducing more from 
what is already there. 

7. In test protocol, Logical Reasoning involves developing a way of thinking 
and/or providing your own argument, opinion, or evaluation based on 
question. 

8. We are eliminating "X" selection from protocols. Please make some decision 
for each blank. 
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